Symbol grouping and recognition in expression recognition

ABSTRACT

A mechanism for recognizing and inputting handwritten mathematical expressions into a computer by providing a part of a multi-path framework is described. The part of the multi-path framework includes a symbol grouping and recognition component that is designed to group input strokes that correspond to a handwritten mathematical expression into a symbol and to recognize the symbol based upon information associated with the grouped input strokes. A method for grouping and recognizing symbols of a handwritten mathematical expression includes receiving a plurality of input strokes corresponding to a handwritten mathematical expression, grouping the plurality of input strokes into symbols, recognizing the symbols based upon information, such as shape and time series information, associated with the grouped input strokes. Intra-group and inter-group information associated with the plurality of input strokes may be utilized to group the input strokes.

This application claims priority to and the benefit of U.S. ProvisionalApplication No. 60/611,847, filed Sep. 22, 2004, which is hereinincorporated by reference.

BACKGROUND

When writing scientific literature and articles using a computer, usersoften must input various and sometimes complex mathematical expressions.Today, a user has to input the mathematical expressions in an indirectmanner. For example, FIGS. 1A and 1B show two systems for inputting amathematical expression: a structured expression editor, such asEquation Editor by Microsoft® Corporation of Redmond, Wash. (FIG. 1A);and an expression descript language, such as L_(A)T^(E)X Equation Editor(FIG. 1B). FIG. 1A shows a large tool box 101 which contains items105-1-105-N corresponding to various mathematical symbols andstructures. The input of expressions may be laborious for some as a userhas to find the proper symbol or structure from the groupings. FIG. 1Bshows a second system that is oriented more towards an expert in thearea of mathematical expression script languages. Users have to becomeexperts of the script language before they may utilize the language toinput expressions freely. Both systems are designed for the mathematicalexpression to be inputted by a keyboard.

The use of an electronic pen and/or stylus input device is a morenatural method for users to input mathematical expressions. The tabletstyle computer allows a user to enter handwritten notes; however,mathematical expressions have not been recognized with high accuracy byexisting handwriting recognition software packages. A need exists forhandwritten mathematical expression recognition to enable pen-basedinput. Comparing to printed expressions, more ambiguities exist inhandwritten expressions. Firstly, it is hard to differentiate symbolsfrom each other just by using shape information. For example, ‘X’ isvery similar to ‘x’, such as for designating a multiplication operation.Another typical example is a ‘dot’. When a dot is located at a positionof a subscript, it is a decimal dot. However, when the dot is at amid-level position, it is a dot operator. Secondly, there are manyuncertainties in a layout structure. For example, a numerator may expandto a region outside of a fraction line because there is not enough roomabove the line.

With the rise in use of the tablet style computer, applications arebeing created and/or updated to implement handwritten annotationrecognition. However, handwritten text recognition and ink documentanalysis are the only recognition and analysis systems enabled in afreehand input system. Handwritten mathematical expression recognitionhas not been available yet.

SUMMARY

Handwritten notations systems for text input allows a user to freelywrite notes. However, conventional systems do not allow a user to inputhandwritten mathematical expressions. The invention is a mechanism forrecognizing and inputting handwritten mathematical expressions into acomputer by providing part of a multi-path framework. The part of themulti-path framework includes a symbol grouping and recognitioncomponent. Claims are directed to systems for receiving and groupinginput strokes corresponding to handwritten mathematical expressions intosymbols and recognizing the symbols based upon information, such asshape and time series information, associated with the grouped inputstrokes. Spatial relationship information with respect to multiplesymbols may be utilized to group the input strokes.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the present invention and theadvantages thereof may be acquired by referring to the followingdescription in consideration of the accompanying drawings, in which likereference numbers indicate like features, and wherein:

FIG. 1A shows a conventional graphical user interface for inputtingmathematical expressions;

FIG. 1B shows another conventional graphical user interface forinputting mathematical expressions;

FIG. 2A illustrates a schematic diagram of a general-purpose digitalcomputing environment in which certain aspects of the present inventionmay be implemented;

FIGS. 2B through 2M show a general-purpose computer environmentsupporting one or more aspects of the present invention;

FIG. 3 is an illustrative block diagram of a framework for a system torecognize handwritten mathematical expressions in accordance with atleast one aspect of the present invention;

FIG. 4 is an example handwritten mathematical expression;

FIG. 5 is an illustrative flowchart showing the sequence of strokedeterminations for recognizing handwritten mathematical expressions inaccordance with at least one aspect of the present invention;

FIG. 6 is an illustrative block diagram of an example baseline structuretree (BST) for representing a mathematical expression in accordance withat least one aspect of the present invention;

FIG. 7 is an illustrative block diagram of an example portion of a BSTfor representing a mathematical expression in accordance with at leastone aspect of the present invention;

FIG. 8 is an illustrative block diagram of an example collection of inkstrokes for the initial creation of a parse tree in accordance with atleast one aspect of the present invention;

FIG. 9 is an illustrative block diagram of an example parse tree aftersymbol grouping and recognition solutions have been determined inaccordance with at least one aspect of the present invention;

FIG. 10 is an illustrative block diagram of an example parse tree aftersubordinate sub-expression analysis solutions have been determined inaccordance with at least one aspect of the present invention;

FIG. 11 is an illustrative block diagram of an example parse tree aftersubscript/superscript and character determination solutions have beendetermined on one of the solution from FIG. 10 in accordance with atleast one aspect of the present invention;

FIG. 12 is an illustrative block diagram of an example semantic treeusing BST symbol and relational nodes in accordance with at least oneaspect of the present invention;

FIG. 13 is an illustrative flowchart of symbol grouping and recognitionin accordance with at least one aspect of the present invention;

FIG. 14 is an illustrative block diagram of an example of a linearstroke sequence from a parse tree in accordance with at least one aspectof the present invention;

FIG. 15 is an illustrative block diagram of an example of a parse treechange after symbol grouping and recognition in accordance with at leastone aspect of the present invention;

FIG. 16 is an illustrative flowchart of symbol recognition in accordancewith at least one aspect of the present invention;

FIG. 17 illustrates an image and writing direction of a point t inaccordance with at least one aspect of the present invention;

FIG. 18 illustrates a quantified directional graph in accordance with atleast one aspect of the present invention;

FIG. 19 illustrates a graph of dominant points in accordance with atleast one aspect of the present invention;

FIG. 20 illustrates a curvature direction of a point t graph inaccordance with at least one aspect of the present invention;

FIG. 21 illustrates a use of features for determining grouping inaccordance with at least one aspect of the present invention;

FIG. 22 illustrates an ambiguous handwritten mathematical expression forsymbol grouping;

FIG. 23 illustrates an optimal segmentation of a sequence in accordancewith at least one aspect of the present invention;

FIG. 24 illustrates examples of non-symbol recognition in accordancewith at least one aspect of the present invention;

FIG. 25 is an illustrative flowchart of symbol training in accordancewith at least one aspect of the present invention;

FIG. 26 is an illustrative flowchart of symbol recognition in accordancewith at least one aspect of the present invention;

FIG. 27 illustrates an inter-symbol spatial relationship configurationin accordance with at least one aspect of the present invention;

FIG. 28 illustrates an intra-symbol spatial relationship configurationin accordance with at least one aspect of the present invention;

FIGS. 29A-29D illustrate examples of handwritten mathematical matrices;

FIGS. 30A-30B illustrate examples of handwritten mathematical multi-lineexpressions;

FIG. 31 illustrates the use of an X-Y projection on a handwrittenmathematical expression in accordance with at least one aspect of thepresent invention;

FIG. 32 illustrates another use of an X-Y projection on a handwrittenmathematical expression in accordance with at least one aspect of thepresent invention;

FIG. 33 illustrates an example of an expression with a tree ofsub-expressions in accordance with at least one aspect of the presentinvention;

FIG. 34 illustrates an example of a rectangle centered control regionfor ‘Above’ and ‘Below’ sub-expression types respectively in accordancewith at least one aspect of the present invention;

FIG. 35 is an equation used to calculate a relational score inaccordance with at least one aspect of the present invention;

FIG. 36 is an illustrative example of a graphical description of theequation from FIG. 35 in accordance with at least one aspect of thepresent invention;

FIG. 37 illustrates an example of a rectangle centered control regionfor ‘Above’ and ‘Below’ sub-expression types respectively in accordancewith at least one aspect of the present invention;

FIG. 38 is an equation used to calculate a relational score inaccordance with at least one aspect of the present invention;

FIG. 39 is an illustrative example of a graphical description of theequation from FIG. 38 in accordance with at least one aspect of thepresent invention;

FIG. 40 illustrates a set of handwritten mathematical expressions inaccordance with at least one aspect of the present invention;

FIG. 41 illustrates relational scores between strokes of handwrittenmathematical expressions in accordance with at least one aspect of thepresent invention;

FIG. 42 illustrates relational scores between strokes of handwrittenmathematical expressions in accordance with at least one aspect of thepresent invention;

FIG. 43 is an overall formal equation to be used to adjust relationalscores by global information in accordance with at least one aspect ofthe present invention;

FIG. 44 is an example relational graph in accordance with at least oneaspect of the present invention;

FIG. 45 is the search process and FIG. 46 is the input and output of thesearch process in accordance with at least one aspect of the presentinvention;

FIG. 47 is a flowchart to illustrate the process of subordinatesub-expression analysis in accordance with at least one aspect of thepresent invention;

FIG. 48 is an illustrative block diagram of an example of a parse treechange after symbol grouping and recognition in accordance with at leastone aspect of the present invention;

FIG. 49 is an illustrative block diagram of an example of a parse treechange after subordinate sub-expression analysis in accordance with atleast one aspect of the present invention;

FIG. 50 is an illustrative flowchart of subscript/superscript andcharacter determination component in accordance with at least one aspectof the present invention;

FIG. 51 is an illustrative graph including scores calculated for givenedges in accordance with at least one aspect of the present invention;

FIG. 52 is an equation to calculate an edge score in accordance with atleast one aspect of the present invention;

FIG. 53 is an illustrative example of score calculators for given partsof handwritten mathematical expression in accordance with at least oneaspect of the present invention;

FIG. 54 is an equation for calculating an offset in the verticaldirection in accordance with at least one aspect of the presentinvention;

FIG. 55 is an equation for calculating a space score in accordance withat least one aspect of the present invention;

FIG. 56 is an equation for calculating a bi-gram probability inaccordance with at least one aspect of the present invention;

FIG. 57 is an illustrative block diagram for a process for removinginvalid spanning trees in accordance with at least one aspect of thepresent invention;

FIG. 58 is an illustrative block diagram of an example of a parse treechange after subscript/superscript and character determination inaccordance with at least one aspect of the present invention;

FIG. 59 is an illustrative block diagram of an example of a parse treechanged after subordinate sub-expression analysis in accordance with atleast one aspect of the present invention;

FIG. 60 is an illustrative graphical user interface for inputtingmathematical expressions in handwritten form in accordance with at leastone aspect of the present invention;

FIG. 61 is an illustrative example of different subordinatesub-expression analysis schemes in accordance with at least one aspectof the present invention;

FIG. 62 illustrates the interaction of a sub expression in amathematical expression input panel in accordance with at least oneaspect of the present invention;

FIG. 63 illustrates an example of a candidate menu in accordance with atleast one aspect of the present invention;

FIG. 64 illustrates an example of a symbol candidate menu in the userinterface of a mathematical expression input panel in accordance with atleast one aspect of the present invention;

FIG. 65 illustrates another example of a candidate menu in accordancewith at least one aspect of the present invention;

FIG. 66 illustrates an example of the input area of a user interface ofa mathematical expression input panel in accordance with at least oneaspect of the present invention;

FIG. 67 illustrates another example of the user interface of amathematical expression input panel in accordance with at least oneaspect of the present invention;

FIG. 68 illustrates another example of the user interface of amathematical expression input panel in accordance with at least oneaspect of the present invention;

FIG. 69 illustrates another example of the user interface of amathematical expression input panel in accordance with at least oneaspect of the present invention;

FIG. 70 illustrates another example of the user interface of amathematical expression input panel in accordance with at least oneaspect of the present invention;

FIG. 71 illustrates another example of the user interface of amathematical expression input panel with an accurate result expressionin accordance with at least one aspect of the present invention;

FIG. 72 is another illustrative graphical user interface for inputtingmathematical expressions in handwritten form in accordance with at leastone aspect of the present invention;

FIG. 73 is the example graphical user interface shown in FIG. 72 with ahandwritten mathematical expression in a handwriting area in accordancewith at least one aspect of the present invention;

FIG. 74 illustrates another example of the graphical user interfaceshown in FIG. 73 with a progress bar in accordance with at least oneaspect of the present invention;

FIG. 75 illustrates another example of the graphical user interfaceshown in FIG. 73 with a result display area in accordance with at leastone aspect of the present invention;

FIG. 76 illustrates another example of the graphical user interfaceshown in FIG. 75 with a candidate list in accordance with at least oneaspect of the present invention;

FIG. 77 illustrates another example of a graphical user interface with adropdown menu for a single character in accordance with at least oneaspect of the present invention;

FIG. 78 illustrates another example of a graphical user interface withplaceholders for a symbol in accordance with at least one aspect of thepresent invention;

FIG. 79 illustrates another example of a graphical user interface with adropdown menu for a sub-expression in accordance with at least oneaspect of the present invention;

FIG. 80 illustrates another example of a graphical user interface with adropdown menu in accordance with at least one aspect of the presentinvention;

FIG. 81 illustrates another example of a graphical user interface withdrag and drop capabilities in accordance with at least one aspect of thepresent invention;

FIG. 82 illustrates another example of a graphical user interface withdrag and drop capabilities in accordance with at least one aspect of thepresent invention;

FIG. 83 illustrates another example of a graphical user interface withdrag and drop capabilities in accordance with at least one aspect of thepresent invention;

FIG. 84 illustrates another example of a graphical user interface of asymbol picker in accordance with at least one aspect of the presentinvention; and

FIG. 85 illustrates another example of a graphical user interface withplaceholders for a symbol in accordance with at least one aspect of thepresent invention.

DETAILED DESCRIPTION

In the following description of the various embodiments, reference ismade to the accompanying drawings, which form a part hereof, and inwhich is shown by way of illustration of various embodiments in whichthe invention may be practiced. It is to be understood that otherembodiments may be utilized and structural and functional modificationsmay be made without departing from the scope of the present invention.

FIG. 2A illustrates an example of a suitable computing systemenvironment 200 on which the invention may be implemented. The computingsystem environment 200 is only one example of a suitable computingenvironment and is not intended to suggest any limitation as to thescope of use or functionality of the invention. Neither should thecomputing system environment 200 be interpreted as having any dependencynor requirement relating to any one or combination of componentsillustrated in the example computing system environment 200.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-heldor laptop devices, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, distributed computing environmentsthat include any of the above systems or devices, and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

With reference to FIG. 2A, an illustrative system for implementing theinvention includes a general-purpose computing device in the form of acomputer 210. Components of computer 210 may include, but are notlimited to, a processing unit 220, a system memory 230, and a system bus221 that couples various system components including the system memoryto the processing unit 220. The system bus 221 may be any of severaltypes of bus structures including a memory bus or memory controller, aperipheral bus, and a local bus using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicsStandards Association (VESA) local bus, and Peripheral ComponentInterconnect (PCI) bus also known as Mezzanine bus.

Computer 210 typically includes a variety of computer readable media.Computer readable media may be any available media that may be accessedby computer 210 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules or other data. Computerstorage media includes, but is not limited to, random access memory(RAM), read only memory (ROM), electronically erasable programmable readonly memory (EEPROM), flash memory or other memory technology, CD-ROM,digital versatile disks (DVD) or other optical disk storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which may be used to store thedesired information and which may accessed by computer 210.Communication media typically embodies computer readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and other wireless media. Combinations of the anyof the above should also be included within the scope of computerreadable media.

The system memory 230 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as ROM 231 and RAM 232. A basicinput/output system 233 (BIOS), containing the basic routines that helpto transfer information between elements within computer 210, such asduring start-up, is typically stored in ROM 231. RAM 232 typicallycontains data and/or program modules that are immediately accessible toand/or presently being operated on by processing unit 220. By way ofexample, and not limitation, FIG. 2A illustrates operating system 234,application programs 235, other program modules 236, and program data237.

The computer 210 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 2A illustrates a hard disk drive 241 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 251that reads from or writes to a removable, nonvolatile magnetic disk 252,and an optical disc drive 255 that reads from or writes to a removable,nonvolatile optical disc 256 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that may be used in the illustrative operating environmentinclude, but are not limited to, magnetic tape cassettes, flash memorycards, digital versatile disks, digital video tape, solid state RAM,solid state ROM, and the like. The hard disk drive 241 is typicallyconnected to the system bus 221 through a non-removable memory interfacesuch as interface 240, and magnetic disk drive 251 and optical discdrive 255 are typically connected to the system bus 221 by a removablememory interface, such as interface 250.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 2A, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 210. In FIG. 2A, for example, hard disk drive 241 isillustrated as storing operating system 244, application programs 245,other program modules 246, and program data 247. Note that thesecomponents may either be the same as or different from operating system234, application programs 235, other program modules 236, and programdata 237. Operating system 244, application programs 245, other programmodules 246, and program data 247 are given different numbers here toillustrate that, at a minimum, they are different copies. A user mayenter commands and information into the computer 210 through inputdevices such as a digital camera (not shown), a keyboard 262, andpointing device 261, commonly referred to as a mouse, trackball or touchpad. Other input devices (not shown) may include a microphone, joystick,game pad, satellite dish, scanner, or the like. These and other inputdevices are often connected to the processing unit 220 through a userinput interface 260 that is coupled to the system bus 221, but may beconnected by other interface and bus structures, such as a parallelport, game port or a universal serial bus (USB). A monitor 291 or othertype of display device is also connected to the system bus 221 via aninterface, such as a video interface 290. In addition to the monitor,computers may also include other peripheral output devices such asspeakers 297 and printer 296, which may be connected through an outputperipheral interface 295.

In one embodiment, a pen digitizer 263 and accompanying pen or stylus264 are provided in order to digitally capture freehand input. Althougha direct connection between the pen digitizer 263 and the user inputinterface 260 is shown, in practice, the pen digitizer 263 may becoupled to the processing unit 220 directly, via a parallel port orother interface and the system bus 221 as known in the art. Furthermore,although the digitizer 263 is shown apart from the monitor 291, theusable input area of the digitizer 263 may be co-extensive with thedisplay area of the monitor 291. Further still, the digitizer 263 may beintegrated in the monitor 291, or may exist as a separate deviceoverlaying or otherwise appended to the monitor 291.

The computer 210 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer280. The remote computer 280 may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above relativeto the computer 210, although only a memory storage device 281 has beenillustrated in FIG. 2A. The logical connections depicted in FIG. 2Ainclude a local area network (LAN) 271 and a wide area network (WAN)273, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks, intranetsand the Internet.

When used in a LAN networking environment, the computer 210 is connectedto the LAN 271 through a network interface or adapter 270. When used ina WAN networking environment, the computer 210 typically includes amodem 272 or other means for establishing communications over the WAN273, such as the Internet. The modem 272, which may be internal orexternal, may be connected to the system bus 221 via the user inputinterface 260, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 210, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 2A illustrates remoteapplication programs 285 as residing on memory device 281. It will beappreciated that the network connections shown are illustrative andother means of establishing a communications link between the computersmay be used.

It will be appreciated that the network connections shown areillustrative and other means of establishing a communications linkbetween the computers may be used. The existence of any of variouswell-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the like ispresumed, and the system may be operated in a client-serverconfiguration to permit a user to retrieve web pages from a web-basedserver. Any of various conventional web browsers may be used to displayand manipulate data on web pages.

A programming interface (or more simply, interface) may be viewed as anymechanism, process, protocol for enabling one or more segment(s) of codeto communicate with or access the functionality provided by one or moreother segment(s) of code. Alternatively, a programming interface may beviewed as one or more mechanism(s), method(s), function call(s),module(s), object(s), etc. of a component of a system capable ofcommunicative coupling to one or more mechanism(s), method(s), functioncall(s), module(s), etc. of other component(s). The term “segment ofcode” in the preceding sentence is intended to include one or moreinstructions or lines of code, and includes, e.g., code modules,objects, subroutines, functions, and so on, regardless of theterminology applied or whether the code segments are separatelycompiled, or whether the code segments are provided as source,intermediate, or object code, whether the code segments are utilized ina runtime system or process, or whether they are located on the same ordifferent machines or distributed across multiple machines, or whetherthe functionality represented by the segments of code are implementedwholly in software, wholly in hardware, or a combination of hardware andsoftware.

Notionally, a programming interface may be viewed generically, as shownin FIG. 2B or FIG. 2C. FIG. 2B illustrates an interface Interface1 as aconduit through which first and second code segments communicate. FIG.2C illustrates an interface as comprising interface objects I1 and I2(which may or may not be part of the first and second code segments),which enable first and second code segments of a system to communicatevia medium M. In the view of FIG. 2C, one may consider interface objectsI1 and I2 as separate interfaces of the same system and one may alsoconsider that objects 11 and 12 plus medium M comprise the interface.Although FIGS. 2B and 2C show bidirectional flow and interfaces on eachside of the flow, certain implementations may only have information flowin one direction (or no information flow as described below) or may onlyhave an interface object on one side. By way of example, and notlimitation, terms such as application programming interface (API), entrypoint, method, function, subroutine, remote procedure call, andcomponent object model (COM) interface, are encompassed within thedefinition of programming interface.

Aspects of such a programming interface may include the method wherebythe first code segment transmits information (where “information” isused in its broadest sense and includes data, commands, requests, etc.)to the second code segment; the method whereby the second code segmentreceives the information; and the structure, sequence, syntax,organization, schema, timing and content of the information. In thisregard, the underlying transport medium itself may be unimportant to theoperation of the interface, whether the medium be wired or wireless, ora combination of both, as long as the information is transported in themanner defined by the interface. In certain situations, information maynot be passed in one or both directions in the conventional sense, asthe information transfer may be either via another mechanism (e.g.information placed in a buffer, file, etc. separate from informationflow between the code segments) or non-existent, as when one codesegment simply accesses functionality performed by a second codesegment. Any or all of these aspects may be important in a givensituation, e.g., depending on whether the code segments are part of asystem in a loosely coupled or tightly coupled configuration, and sothis list should be considered illustrative and non-limiting.

This notion of a programming interface is known to those skilled in theart and is clear from the foregoing detailed description of theinvention. There are, however, other ways to implement a programminginterface, and, unless expressly excluded, these too are intended to beencompassed by the claims set forth at the end of this specification.Such other ways may appear to be more sophisticated or complex than thesimplistic view of FIGS. 2B and 2C, but they nonetheless perform asimilar function to accomplish the same overall result. We will nowbriefly describe some illustrative alternative implementations of aprogramming interface.

A. Factoring

A communication from one code segment to another may be accomplishedindirectly by breaking the communication into multiple discretecommunications. This is depicted schematically in FIGS. 2D and 2E. Asshown, some interfaces may be described in terms of divisible sets offunctionality. Thus, the interface functionality of FIGS. 2B and 2C maybe factored to achieve the same result, just as one may mathematicallyprovide 24, or 2 times 2 times 3 times 2. Accordingly, as illustrated inFIG. 2D, the function provided by interface Interface1 may be subdividedto convert the communications of the interface into multiple interfacesInterface1A, Interface1B, Interface1C, etc. while achieving the sameresult. As illustrated in FIG. 2E, the function provided by interface I1may be subdivided into multiple interfaces I1 a, I1 b, I1 c, etc. whileachieving the same result. Similarly, interface I2 of the second codesegment which receives information from the first code segment may befactored into multiple interfaces I2 a, I2 b, I2 c, etc. When factoring,the number of interfaces included with the 1st code segment need notmatch the number of interfaces included with the 2nd code segment. Ineither of the cases of FIGS. 2D and 2E, the functional spirit ofinterfaces Interface1 and I1 remain the same as with FIGS. 2B and 2C,respectively. The factoring of interfaces may also follow associative,commutative, and other mathematical properties such that the factoringmay be difficult to recognize. For instance, ordering of operations maybe unimportant, and consequently, a function carried out by an interfacemay be carried out well in advance of reaching the interface, by anotherpiece of code or interface, or performed by a separate component of thesystem. Moreover, one of ordinary skill in the programming arts mayappreciate that there are a variety of ways of making different functioncalls that achieve the same result.

B. Redefinition

In some cases, it may be possible to ignore, add or redefine certainaspects (e.g., parameters) of a programming interface while stillaccomplishing the intended result. This is illustrated in FIGS. 2F and2G. For example, assume interface Interface1 of FIG. 2B includes afunction call Square (input, precision, output), a call that includesthree parameters, input, precision and output, and which is issued fromthe 1st Code Segment to the 2nd Code Segment. If the middle parameterprecision is of no concern in a given scenario, as shown in FIG. 2F, itcould just as well be ignored or even replaced with a meaningless (inthis situation) parameter. One may also add an additional parameter ofno concern. In either event, the functionality of square may beachieved, so long as output is returned after input is squared by thesecond code segment. Precision may very well be a meaningful parameterto some downstream or other portion of the computing system; however,once it is recognized that precision is not necessary for the narrowpurpose of calculating the square, it may be replaced or ignored. Forexample, instead of passing a valid precision value, a meaningless valuesuch as a birth date could be passed without adversely affecting theresult. Similarly, as shown in FIG. 2G, interface I1 is replaced byinterface I1′, redefined to ignore or add parameters to the interface.Interface I2 may similarly be redefined as interface I2′, redefined toignore unnecessary parameters, or parameters that may be processedelsewhere. The point here is that in some cases a programming interfacemay include aspects, such as parameters, which are not needed for somepurpose, and so they may be ignored or redefined, or processed elsewherefor other purposes.

C. Inline Coding

It may also be feasible to merge some or all of the functionality of twoseparate code modules such that the “interface” between them changesform. For example, the functionality of FIGS. 2B and 2C may be convertedto the functionality of FIGS. 2H and 21, respectively. In FIG. 2H, theprevious 1st and 2nd Code Segments of FIG. 2B are merged into a modulecontaining both of them. In this case, the code segments may still becommunicating with each other but the interface may be adapted to a formwhich is more suitable to the single module. Thus, for example, formalCall and Return statements may no longer be necessary, but similarprocessing or response(s) pursuant to interface Interface1 may still bein effect. Similarly, shown in FIG. 21, part (or all) of interface I2from FIG. 2C may be written inline into interface I1 to form interfaceI1″. As illustrated, interface I2 is divided into I2 a and I2 b, andinterface portion I2 a has been coded in-line with interface I1 to forminterface I1″. For a concrete example, consider that the interface I1from FIG. 2C performs a function call square (input, output), which isreceived by interface I2, which after processing the value passed withinput (to square it) by the second code segment, passes back the squaredresult with output. In such a case, the processing performed by thesecond code segment (squaring input) may be performed by the first codesegment without a call to the interface.

D. Divorce

A communication from one code segment to another may be accomplishedindirectly by breaking the communication into multiple discretecommunications. This is depicted schematically in FIGS. 2J and 2K. Asshown in FIG. 2J, one or more piece(s) of middleware (DivorceInterface(s), since they divorce functionality and/or interfacefunctions from the original interface) are provided to convert thecommunications on the first interface, Interface1, to conform them to adifferent interface, in this case interfaces Interface2A, Interface2Band Interface2C. This might be done, e.g., where there is an installedbase of applications designed to communicate with, say, an operatingsystem in accordance with an Interface1 protocol, but then the operatingsystem is changed to use a different interface, in this case interfacesInterface2A, Interface2B and Interface2C. The point is that the originalinterface used by the 2nd Code Segment is changed such that it is nolonger compatible with the interface used by the 1st Code Segment, andso an intermediary is used to make the old and new interfacescompatible. Similarly, as shown in FIG. 2K, a third code segment may beintroduced with divorce interface DI1 to receive the communications frominterface I1 and with divorce interface DI2 to transmit the interfacefunctionality to, for example, interfaces I2 a and I2 b, redesigned towork with DI2, but to provide the same functional result. Similarly, DI1and DI2 may work together to translate the functionality of interfaces11 and 12 of FIG. 2C to a new operating system, while providing the sameor similar functional result.

E. Rewriting

Yet another possible variant is to dynamically rewrite the code toreplace the interface functionality with something else but whichachieves the same overall result. For example, there may be a system inwhich a code segment presented in an intermediate language (e.g.Microsoft IL, Java ByteCode, etc.) is provided to a Just-in-Time (JIT)compiler or interpreter in an execution environment (such as thatprovided by the .Net framework, the Java runtime environment, or othersimilar runtime type environments). The JIT compiler may be written soas to dynamically convert the communications from the 1st Code Segmentto the 2nd Code Segment, i.e., to conform them to a different interfaceas may be required by the 2nd Code Segment (either the original or adifferent 2nd Code Segment). This is depicted in FIGS. 2L and 2M. As canbe seen in FIG. 2L, this approach is similar to the Divorce scenariodescribed above. It might be done, e.g., where an installed base ofapplications are designed to communicate with an operating system inaccordance with an Interface1 protocol, but then the operating system ischanged to use a different interface. The JIT Compiler could be used toconform the communications on the fly from the installed-baseapplications to the new interface of the operating system. As depictedin FIG. 2M, this approach of dynamically rewriting the interface(s) maybe applied to dynamically factor, or otherwise alter the interface(s) aswell.

It is also noted that the above-described scenarios for achieving thesame or similar result as an interface via alternative embodiments mayalso be combined in various ways, serially and/or in parallel, or withother intervening code. Thus, the alternative embodiments presentedabove are not mutually exclusive and may be mixed, matched and combinedto produce the same or equivalent scenarios to the generic scenariospresented in FIGS. 2B and 2C. It is also noted that, as with mostprogramming constructs, there are other similar ways of achieving thesame or similar functionality of an interface which may not be describedherein, but nonetheless are represented by the spirit and scope of theinvention, i.e., it is noted that it is at least partly thefunctionality represented by, and the advantageous results enabled by,an interface that underlie the value of an interface.

Handwritten mathematical expression recognition is needed to enablepen-based input of mathematical expressions. Aspects of the presentinvention propose a framework for handwritten mathematical expressionrecognition, which may output multiple expression candidates. Inaccordance with one embodiment, the multi-path framework utilizesmulti-path algorithms and outputs multiple results in severalcomponents, including symbol grouping and recognition, tabular structureanalysis, subordinate sub-expression analysis, and subscript/superscriptanalysis, and character determination. The system may output multiplerecognition candidates for each handwritten expression by combiningmultiple results from the various components. With a correction userinterface (UI), a user may select a proper choice from the candidatessupplied by the system. Aspects of the present invention enable morenatural input of mathematical expressions.

Definitions

Stroke is a trajectory of a pen tip between a pen down position and apen up position. A stroke may be described by a series of points withtimestamps (x, y, time).

Symbol includes of one or multiple strokes. A symbol is a handwrittenversion of pre-defined mathematical characters including Latinalphabets, digits, Greek letters, etc.

Expression is a meaningful combination of mathematical symbols.

Character is the corresponding computer code of a handwritten symbol.Symbol recognition takes the strokes of a symbol as input and outputsthe corresponding character of the symbol.

Dominant symbol is a mathematical symbol that may be attached tosubordinate sub-expressions. The spatial relationships between dominantsymbols and its sub-expressions are variants to the dominant symbols'types. A description of relational types is described below under thesection entitled, “Subordinate Sub-expression Analysis”.

Sub-expression is a meaningful sub-part of an expression. An expressionmay include several sub-expressions, which form a tree structureaccording to their relationships of principal and subordinate. Asub-expression is an expression. There are two kinds of sub-expressions.A subordinate sub-expression is a sub-expression subordinate to adominant symbol. Subscript and superscript sub-expressions aresub-expressions that are a subscript or superscript of another symbol.

BST tree (baseline structure tree) is a data structure for representingan expression. In the representation, an expression is a tree, whoselevels are baselines. Baseline means that symbols within a baseline arelocated in a horizontal line. Here, a baseline is a synonym ofsub-expression.

Parse tree is an extended version of a BST tree. A parse tree may storemultiple results for components of the system and support thefunctionality of providing multiple recognized candidates for ahandwritten expression. A parse tree may be included within a datastructure for a computer-readable medium.

Symbol recognizer is the model that implements symbol recognition. Thesymbol recognizer analyzes all available information, such as shape andtime series information of a symbol to recognize the symbols.

On-line features are features that use time series information. Usually,a stroke has the time information of each point of the stroke.

Off-line features do not use time series information, instead they useshape information. Off-line features in symbol recognition are oftenextracted based on image and pixels.

The Gaussian Mixture Model (GMM) is a mixture probability distributionmodel. A GMM is a linear combination of K Gaussian components.

Tabular structure includes matrix and multi-line expression. Bothstructures may be divided into rows and/or columns. Multi-lineexpressions always have only one column and have only one curly bracketon the left side. Matrices have brackets on both the left and rightsides.

A matrix is a group of structured strokes that may be divided intomultiple rows and/or columns and surrounded by a pair of brackets atboth the left and right sides (FIGS. 29A-D). Column vectors (FIG. 29D)and determinants (FIG. 29C) are also regarded as special matrices.

A multi-line expression is a group of strokes that may be divided intoseveral left aligned rows which are led by a left curly bracket (FIGS.30A-30B).

Brackets in tabular structure analysis are a group of symbols thatencapsulate tabular structures. For matrices, it may be a bracket, asquare bracket or a vertical line at both the left and right side. Formulti-line expression, it may be curly bracket at the left side.

Multi-Path Framework Overview

FIG. 3 is an illustrative block diagram of a framework 300 for a systemto recognize handwritten mathematical expressions in accordance with atleast one aspect of the present invention. The framework may includefive components: a symbol grouping and recognition component 303, atabular structure analysis component 305, a subordinate sub-expressionanalysis component 307, a subscript, superscript analysis and characterdetermination component 309, and a semantic structure analysis component311. Among these components, the symbol grouping and recognitioncomponent 303, the subordinate sub-expression analysis component 307,and the subscript, superscript analysis and character determinationcomponent 309 may output multiple results.

The symbol grouping and recognition component 303 receives a handwrittenmathematical expression 301 and is responsible for grouping strokes intosymbols and for recognizing the symbols. The output of component 303 ishow strokes are grouped to symbols, and possible character candidatesand corresponding confidences for each symbol.

Compared to plain text, mathematical expression is a more complexstructured layout. Expressions have specific structures. For example, anN-array summation (‘I’) has two attached sub-expressions, above andbelow sub-expressions to express below and above summation limits. Also,subscripts and superscripts are typical structures in expressions. Asidefrom hierarchical structures, tabular expression is high levelstructure, where multiple sub-expressions are at the same level forminga table. Such structure information is useful for recognizingexpressions. The structure analysis component is configured to determinestructure information. In accordance with at least one aspect of thepresent invention, the structure analysis component includes thefollowing three sub-components: the tabular structure analysis component305; the subordinate sub-expression analysis component 307; and thesubscript, superscript analysis and character determination component309.

Tabular structure analysis component 305 includes matrix andmultiple-line structure recognition. Tabular structure analysiscomponent 305 identifies each table and the content of each cell in eachtable. After tabular structure analysis component 305 identifies tabularstructure, later structure analysis components regard each cell as asub-expression and analyze the structure for each cell further.Subordinate sub-expression analysis component 307 is used to findsubordinate sub-expressions for each dominant symbol. Subscript,superscript analysis and character determination component 309 findssubscript and superscript structures and decides each symbol's finalcharacter at the same time.

After being processed by the structure analysis component, a treestructure of sub-expressions is found, and the characters of the symbolsare decided for the input handwritten mathematical expression. However,the inherent semantic structure is not yet represented in a datastructure of a parse tree. Therefore, the semantic structure analysiscomponent 311 is used to translate the linear symbols that are in asub-expression into a syntax tree and to adjust the parse tree accordingto the syntax tree, resulting in a recognized mathematical expression313.

FIG. 4 is an example handwritten mathematical expression that is usedherein for illustrative purposes. FIG. 5 is an illustrative flowchartshowing the sequence of stroke determinations for recognizinghandwritten mathematical expressions in accordance with at least oneaspect of the present invention. FIG. 5 shows a multi-path algorithmworkflow with the example from FIG. 4. Inputted handwritten expression501 is a sequence of strokes. At the symbol recognition and groupinglevel 503, two possible grouping results are identified. The firstresult 551 is that the first two strokes are grouped into a symbol ‘k’.The second result 581 is that the first two strokes are separated into asymbol ‘l’ and a symbol ‘<’. At this point, the recognition of the othersymbols is the same. At the tabular structure analysis level 505, theparse tree keeps its structure because there is no tabular structure inthe example. At the subordinate sub-expression analysis level 507,following each of the two grouping results 551 and 581 are also twofeasible results. One result 563 is that the symbols of “b”, “2”, “−”,“4”, “a”, and “c” are a radicand of the radical sign. The second result565 is that the symbols of “b”, “2”, “−”, “4”, and “a” are a radicand ofthe radical sign, but that the symbol of “c” is in the mainsub-expression, i.e., it is not a radicand of the radical sign. At thesubscript, superscript analysis and character determination level 509,there are also two results for the radicand part. For the result 563,candidate “b²−4ac” 577 and candidate “bz−4ac” 579 result. For result565, the candidate results are similar. The system ends with 8reasonable candidates for a relatively simple expression beforeproceeding to the semantic structure analysis component level 511.

In accordance with aspects of the present invention, a data structurestores the multiple results obtained by all the multi-path algorithms.The structure is passed from the first component to the last component.Every component gets the structure from the previous component, performsits analysis operation, and then writes its results back into thestructure, passing the structure to the next component. Afterrecognition is complete, the system gets the data structure savingmultiple results from many components. With the data structure, a wholeexpression's candidate may be determined by selecting a result for eachmulti-path component sequentially. Furthermore, the system may determinemultiple expression candidates with different selections and then rankthe candidates based on a combined score, which may include scores ofvarious components.

Before describing the data structure of multi-results, a data structurerepresenting a single structured expression is described. A baselinestructure tree (BST) is used to represent an expression. One point of aBST tree is to view an expression as a tree including multi-levelbaselines. Within a baseline, symbols are horizontal neighbors. In thelayout, the symbols lie in a horizontal line.

FIG. 6 is an illustrative block diagram of an example baseline structuretree (BST) structure for representing a mathematical expression inaccordance with at least one aspect of the present invention. Theexample includes three levels of baselines. The first baseline“1<radical” is the main baseline. The second baseline “b−4ac” is asub-expression subordinate to the radical sign. The third baseline “2”is a superscript of “b” which is at the second baseline. Four types oftree nodes are defined to represent a BST tree in accordance with thepresent invention. A stroke node, shown by a diamond shape, such as 601,represents a stroke in ink expressions. It stores position (x, y) ofeach point of a stroke and a timestamp when the pen tip is down. Asymbol node, shown by a circle shape, such as 603, represents a symbol,which may include several strokes. It records the references to itschild nodes, which are stroke nodes. A symbol node also stores asymbol's character candidates and confidences determined by symbolrecognition.

A BST symbol node, shown by a rectangular shape, such as 605, is amiddle-level node between a symbol node and a relational node. The BSTsymbol node is child node of a relational node. A BST symbol node mayhave a symbol node and a relational node as its child nodes. A BSTsymbol node is configured to represent a compound of a dominant symboland its sub-baselines (sub-expressions). A single symbol, which has nosub-baselines (sub-expressions), is wrapped into a BST symbol node witha tag “normal” in order to become a child of a relational node. Thefollowing tags are defined for a BST symbol node:

-   -   Normal: a symbol without subordinates.    -   Decorated: a symbol with subscript or superscript.    -   Fraction: a fraction line with denominator and numerator.    -   Radical: a radical sign with radicand.    -   Integral: an integral sign with integral limits.    -   N-Array: an N-Array sign (Σ,Π) with above or below limits.    -   Limits: a symbol “lim” with its below subordinate.    -   Hats: a hat sign (ˆ,−) with its decorated subordinate.

A relational node, shown by a rounded rectangular shape, such as 607,represents a baseline (sub-expression), which includes several BSTsymbol nodes located on a horizontal line. Its children are BST symbolnodes. The following tags are defined for a relational node:

-   -   Above: a sub-expression above fraction line or N-Array sign.    -   Below: a sub-expression below fraction line or N-Array sign.    -   Radicand: a sub-expression that is a radicand of a radical sign.    -   Radical index: a sub-expression that is radical index of a        radical sign.    -   Superscript: a superscript sub-expression.    -   Subscript: a subscript sub-expression.    -   Expression: the main (top-level) sub-expression.

Aside from the above four types of nodes, another type of node, asolution node, is included in the system to represent various resultsfor the same object. FIG. 7 shows how to use solution nodes to representtwo interpretations of strokes. Solution1 means “b²−4ac”, whileSolution2 means “bz−4ac”. As FIG. 7 shows, the two solutions refer tothe same set of strokes. In implementation, it is also necessary toperform these multiple references to the same objects. Because multipleresults may be outputted from three components, duplication of a tree ora sub-tree for each of these results would require a huge amount ofmemory due to exponential combinations. Moreover, the idea of simpleduplication also results in unnecessary repeated calculations for sameobjects. For example, symbols of “b2−4ac” are a sub-expression in theresult 563 from FIG. 5. To implement duplication, subscript, superscriptanalysis, and character determination has to be done at least twice forthe sub-expression because symbols of “b2−4ac” are duplicated for eachway of grouping strokes. As shown in FIG. 5, results 577 and 579 areduplicated for both results 563 and 565. Therefore, in accordance withaspects of the present invention, a data structure is implemented wherethe same child objects may be referred to by multiple parent objects.With this implementation, an extended BST tree is not just a treestructure, but a directional acyclic graph. With such a new type of nodeand design of multiple references, a BST tree may be extended to a datastructure, which may store multiple results obtained by the components.In one embodiment, an extended BST tree is parsed component by componentand is often referred to as a parse tree. A parse tree may be includedwithin a data structure.

The following paragraphs will use the example in FIGS. 4 and 5 toexplain how the parse tree evolves by each component.

Handwritten Expression Input 501

Before recognition starts, the expression is a set of strokes withoutstructure information. As shown in FIG. 8, the system collects all inkstrokes and creates a parse tree, which is a sequence of strokes. Atthis point, all stroke nodes are located under a root expression node,parallel to each other.

Symbol Grouping and Recognition 503

This component takes the input of strokes, and groups the strokes intosymbols using a dynamic programming algorithm. Symbol nodes are createdto store results at this stage. During dynamic programming, a symbolrecognizer is called to test if several strokes could be a meaningfulsymbol. In this component, there are multiple ways to group strokes. Sosolution nodes are created in the parse tree to store the multipleresults. FIG. 9 shows the parse tree after the symbol grouping andrecognition component has determined its results.

FIG. 9 also shows how to use solution nodes to store two ways ofgrouping efficiently in a parse tree. In the example, one way is thatthe first two strokes are separated into a symbol ‘l’ and a symbol ‘<’.The other is that the two strokes are grouped into a symbol ‘k’.Grouping results of the other strokes are the same in both ways. In FIG.9, these same parts, shown in the dashed box, are referred to by twosolution nodes. As such, the data for these parts are only stored as onecopy in a data structure. The different parts, shown in the solid linedbox, are referred to respectively by two solutions. Symbols located inthe different parts also refer to the same stroke nodes. With thisdesign of multiple references, multiple results may be stored in theextended BST tree in a manner that saves memory resources andcomputation time.

The symbol recognizer is called again for each grouped symbol to findpossible character candidates and corresponding confidences. Thecharacter candidates and confidences are stored at corresponding symbolnodes. They will be passed on to the next component. Later a structureanalysis component performs its operation based on the symbol nodeinformation.

Subordinate Sub-Expression Analysis Component 507

This component finds subordinate sub-expressions subordinate to dominantsymbols. The component finds all possible dominant symbol candidates.Then it tentatively looks for subordinate symbols for the candidatesusing spatial information, such as symbol distance, size, etc. Ifsubordinate symbols are found, then the candidate is a real dominantsymbol. Otherwise, the candidate is not a dominant symbol. For each realdominant symbol, the found subordinate symbols construct a subordinatesub-expression.

FIG. 10 shows results of the subordinate sub-expression analysiscomponent. Only the results following one grouping solution node aredisplayed in FIG. 10 because the branch following the other groupingsolution node is similar to this one. As shown, one solution is that thesymbols of “b”, “2”, “−”, “4”, “a”, and “c” construct a sub-expressionsubordinate to the radical sign. The other solution is that the symbolsof “b”, “2”, “−”, “4”, and “a” construct the radicand sub-expression.

Subscript, Superscript Analysis and Character Determination Component509

Subscript and superscript structures are identified in this step.Subscript and superscript structures are not only related to thesymbols' spatial relationship, they are also dependent on the symbols'characters. For example, ‘x’, as used for multiplication operations, cannot be a subscript. Therefore, the component performs two tasks,subscript, superscript analysis and character determination, at the sametime. Moreover, syntax analysis is utilized in the component to verifythat multiple results outputted by this component are valid in the senseof expression grammar.

FIG. 11 shows two results of the component following only one solutionof the subordinate sub-expression analysis component. As shown, thereare two solution nodes for the radicand sub-expression. One solution is“b²−4ac”, the other solution is “bz−4ac”.

Semantic Structure Analysis

After previous processing, a tree structure of sub-expressions is builtup and every character is determined. But semantic structure is notdiscovered in its sub-expressions. In order to make recognizedexpression become a semantic structure, text strings translated fromsub-expressions are parsed by syntax analysis and transformed into asyntax tree. Finally, this component revises the parse tree according tothe results of syntax analysis. The system names the final parse tree asa semantic tree of the expression.

A semantic tree corresponds to the semantic structure of an expression.In the tree, high level math concepts, such as operators, operands, andpriorities etc., are defined. With the semantic tree, the expression maybe calculable. FIG. 12 shows how to represent a semantic tree using BSTsymbol and relational nodes. There, symbol “−” becomes a BST symbol,because it is an operator. This BST symbol has two relational nodes,representing two operands respectively.

The component uses a context-free parser to do syntax analysis. Theparser algorithm is a well-known technique, widely applied in the fieldsof language compiler, natural language processing, knowledge-basedsystem, etc. In the system, a library of grammar rules for mathematicalexpression may be built. The library may include in excess of 1,000grammar rules. The following are three example rules about a fractionstructure:

-   -   FRACTION->FRACTIONLINE DENOMINATOR NUMERATOR    -   DENOMINATOR->SYMBOL_LEFTCONTROL OPERAND SYMBOL_RIGHTCONTROL    -   NUMERATOR->SYMBOL_LEFTCONTROL OPERAND SYMBOL_RIGHTCONTROL

Aspects of the present invention recognize a multitude of symbolsincluding Latin alphabets (a, b, c, A, B, C, etc.), Greek letters (α, β,θ, λ, ω, etc.), Latin digits (1, 2, 3, 4, 5, etc.), Operators (Σ, Π, ∫,±, ×, etc.), and frequently used mathematical symbols (∂, etc.). Aspectsof the present invention also support frequently used expression types,including Arithmetic operations (+, −, ×, /, etc.), Fraction (−),Radical (√), Integral (∫), N-Array (Σ,Π), Limits (lim), multi-letterfunctions (sin, cos, tan, log, In, etc.), Hats (â,{right arrow over(AB)}, etc.), and matrix and/or multi-line expressions. In oneembodiment, in excess of 150 different mathematical symbols may berecognized by the system.

Symbol Grouping and Recognition

Referring back to FIG. 3, the symbol grouping and recognition component303 is one part of the whole mathematical expression recognition system300. The output of component 303 is how strokes are grouped to symbolsand possible character candidates and a corresponding confidence foreach symbol. Symbol grouping groups strokes into math symbols. Symbolrecognition recognizes the symbols using all available information, e.g.shape, time series, and context. As described above, due to ambiguitiesof a symbol, symbol recognition outputs multiple recognition results.

An on-line handwritten symbol written on a digitizing tablet isrepresented as a sequence of strokes, which are the loci of the pen tipfrom its pen-down to pen-up position. On-line recognition isconsiderably different from off-line recognition because of the dynamicinformation on writing. Symbol recognition methods are roughlyclassified into three major groups: statistical method, structure andsyntax analysis method, and model matching methods. In accordance withat least one aspect of the present invention, statistical methods areused to recognize symbols. A statistical symbol recognition methodconsists of two processes, a training stage and a recognition stage. Thetraining framework and recognition framework are shown in FIGS. 25 and26.

In the training stage, a large amount of training data is assumedavailable to build some statistical model. Handwritten strokes are firstsmoothed and normalized to a fixed size. In sequence, some statisticalfeatures are extracted from the unknown symbol. Dimensional reduction isused to optimize these features. Next, Gaussian Mixture Models (GMM) aretrained as a classifier. Then, discriminative training is adopted tooptimize the GMM. In the recognition stage, after preprocessing andfeature extraction, the unknown symbol is classified to the class whosemembers have the most similar features. GMM is a mixture probabilitydistribution model, which provides better similarity measurement thantemplate based classifiers.

Many mathematical symbols are written with multiple strokes. Forinstance, ‘A’ may be written with 3 strokes. Usually, an expressionconsists of several symbols, and each symbol may have one or multiplestrokes. But in the input data, all strokes of the symbols are mixedtogether. Therefore, the first step of handwritten expressionrecognition is to identify which strokes construct a symbol, and howmany symbols are in the handwriting expression. After theidentification, ink strokes are grouped into symbols. Then, a subsequentstructure analysis may perform further recognition based on the new datarepresentation provided by the symbol grouping step.

Symbol grouping and symbol recognition interacts with each other duringthe recognition process. FIG. 13 shows the flowchart of symbol groupingand recognition. Symbol grouping and recognition 1303 receives inputstrokes 1301 from the parse tree (as shown in FIG. 14), and groups inputstrokes 1301 into symbols 1305 using dynamic programming algorithm.During dynamic programming, symbol recognizer is called to test whetherseveral strokes could be a meaningful symbol. In this component, thereare possible multiple results of symbol grouping. Symbol grouping 1323and symbol recognition 1333 create solution nodes in a parse tree towrite back the multiple results. FIG. 15 shows the changed parse treeafter symbol recognition and grouping. Symbol nodes are created at thisstage. Then symbol recognizer is called again for each grouped symbol tofind possible characters candidates and confidences. Charactercandidates and confidence of a grouped symbol are stored bycorresponding symbol nodes. They are passed to succeeding componentsthrough the parse tree. The structure analysis component performs itsoperation at a later time based on the symbol node information.

Symbol Recognition

In one embodiment, an approach based upon Gaussian Mixture Model (GMM)is used to implement symbol recognition. An off-line feature is used inthe GMM based symbol recognition. For computing off-line features, thewriting direction for each point in the symbol strokes is calculated.The writing direction is the tangent direction of a sampling point.Usually, a tangent direction is not easy to calculate. For samplingpoint t, the tangent direction is estimated by using the line directionbetween sampling point t-1 and sampling point t. The angle (a) betweenthis line and the horizontal line is the value of the writing directionof point t. The writing direction is defined by:Δ  x(t) = x(t) − x(t − 1) Δ  y(t) = y(t) − y(t − 1)${\cos\quad{\alpha(t)}} = \frac{\Delta\quad{x(t)}}{\sqrt{{\Delta\quad{x^{2}(t)}} + {\Delta\quad{y^{2}(t)}}}}$${\sin\quad{\alpha(t)}} = \frac{\Delta\quad{y(t)}}{\sqrt{{\Delta\quad{x^{2}(t)}} + {\Delta\quad{y^{2}(t)}}}}$

FIG. 17 shows the writing direction of the t^(th) point in the pointsequence. The point sequence is then converted to an image. Eachadjacent point pair in the point sequence is connected with a line andan image may be obtained. FIG. 17 shows the converted image of β. Thewriting direction of each point on a line is equal to the writingdirection of its former sampling point. In other words, the writingdirection of the points between sampling point t and sampling point t+1is equal to that of sampling point t. The size of the image is the samewith ink strokes.

Because the size of different images is different, it is inconvenient tomeasure them. So, the image was normalized to a fixed scale of 64×64pixels in symbol recognition. In this example, a nonlinear normalizationis used. After normalization, the center of the normalized image shouldcorrespond to the gravity point. Normalization may be expressed as:ax ² +bx+c=mdy ² +ey+f=n,where (x, y) is a point of original image, and (m, n) is a correspondingpoint in a normalized image. Here, five corresponding points may beobtained to solve this equation:(0,0)→(0,0),(0,0)→(0,M),(Y,0)→(N,0),(X,Y)→(M,N),(Centroid)→(M/2,N/2),where X, Y is the width and height of the original image, and M, N iswidth and height of the normalized image, respectively. After these six(6) parameters are calculated, the origin point will be normalized usingthe above two equations. The centroid point may be calculated by:${CX} = \frac{\sum\limits_{i = 1}^{m \times n}\quad{{p(i)}{x(i)}}}{\sum\limits_{i = 1}^{m \times n}\quad{p(i)}}$${CY} = \frac{\sum\limits_{i = 1}^{m \times n}\quad{{p(i)}{y(i)}}}{\sum\limits_{i = 1}^{m \times n}\quad{p(i)}}$if i^(th) is black pixel, p(i)=1, else p(i)=0.

The writing direction of each point is classified to eight (8) levels.FIG. 18 shows the quantified 8 directions. For example, as shown in FIG.18, if the writing direction a of a point is 140°, then the quantifieddirection value is 4.

The commonly used mesh statistical method may be used to obtain afeature vector. The image may be evenly subdivided into 8 rows and 8columns, so that the size of each sub-region is 8×8. The number of eachdirection in each sub-region is counted. A 512-dimension feature vector(8 rows×8 columns×8 directions) is obtained. For example, there are fiveblack pixels in a sub-region. Writing directions of the 5 pixels are30°, 40°, 50°, 80°, 110°. The quantified direction of each pixel, 1, 1,2, 2, 3, may be obtained respectively. The 8 dimensional feature vectorof this sub-region is 2, 2, 1, 0, 0, 0, 0, 0. All 64 sub-regions havesuch an 8 dimensional feature vector and finally a 512-dimensionalfeature vector may be generated.

Dimension reduction is another step in symbol recognition. Two reasonsfor using dimension reduction include cost and relativity. A512-dimension system requires much more in calculation and some featuresmay be correlated to other features, e.g., redundant information existsin the 512-dimension feature. In accordance with at least one aspect ofthe present invention, the 512-dimension feature is transformed to a128-dimension feature. Any of a number of different dimension reductiontechniques may be used for this purpose and those skilled in the artwould understand the various techniques.

A technique commonly used for dimensionality reduction is Fisher'sLinear Discriminant (FLD). It should be understood by those skilled inthe art that FLD is commonly known. FLD is an example of a classspecific method, in the sense that it tries to “shape” the scatter inorder to make it more reliable for classification. This method selectsthe projection W_(opt) in such a way that the ratio of the between-classscatter and the within-scatter is maximized. The between-class scattermatrix may be defined as:${S_{B} = {\sum\limits_{i = 1}^{c}\quad{{P_{i}\left( {\mu_{i} - \mu} \right)}\left( {\mu_{i} - \mu} \right)^{T}}}},$and the within-class scatter matrix may be defined as:${S_{w} = {\sum\limits_{i = 1}^{c}\quad{P_{i}\frac{1}{N_{i}}{\sum\limits_{j = 1}^{N_{i}}\quad{\left( {x_{j} - \mu_{i}} \right)\left( {x_{j} - \mu_{i}} \right)^{T}}}}}},$where μ_(i) is the mean vector of class X_(i), P_(i) is priorprobability of class X_(i), and N_(i) is the number of samples in classX_(i). As a result, the optimal projection W_(opt) is chosen as thematrix with orthonormal columns which maximizes the ratio of thedeterminant of the between-class scatter matrix of the projected samplesto the determinant of the within-class scatter matrix of the projectedsamples,$W_{opt} = {{\arg\quad\max\frac{{W^{T}S_{B}W}}{{W^{T}S_{w}W}}} = \left\lbrack {w_{1}\quad w_{2}\quad\cdots\quad w_{m}} \right\rbrack}$

This ratio is maximized when the column vectors of projection matrix Ware the eigenvectors of S_(w) ⁻¹S_(b) associated with the largest eigenvalues. The result is to maximize the between-class scatter whileminimizing the within-class scatter.

To avoid the ill-pose problem when computing the eigen values of thematrix S_(w) ⁻¹S_(b), one embodiment of the present invention adopts themethod as described in Swets, Daniel L. and Weng, John, UsingDiscriminant Eigen Features for Image Retrieval, IEEE Trans PatternAnalysis and Machine Intelligence, vol. 18, pp. 831-836, 1996. It shouldbe understood by those skilled in the art, this method is merelyillustrative and that any other similar methods may be used for thepurpose.

H and A are computed such that, S_(w)=HΛH^(T), where H is orthogonal andΛ is diagonal. Then,${\left( {H\quad\Lambda^{- \frac{1}{2}}} \right)^{t}{S_{w}\left( {H\quad\Lambda^{- \frac{1}{2}}} \right)}} = I$

U and Σ are then computed such that,${{\left( {H\quad\Lambda^{- \frac{1}{2}}} \right)^{t}{S_{b}\left( {H\quad\Lambda^{- \frac{1}{2}}} \right)}} = {U{\sum\quad U^{t}}}},$where U is orthogonal and Σ is diagonal. Then,S_(w)⁻¹ = H  Λ⁻¹H^(t), and$S_{b} = {H\quad\Lambda^{\frac{1}{2}}U{\sum\quad{U^{t}\Lambda^{\frac{1}{2}}H^{t}}}}$

Defining, ${\nabla{= {H\quad\Lambda^{- \frac{1}{2}}U}}},$the decomposition of S_(w) ⁻¹S_(b) may be found as following:$\begin{matrix}{{S_{w}^{- 1}S_{b}} = {H\quad\Lambda^{- 1}H^{t}H\quad\Lambda^{\frac{1}{2}}U{\sum\quad{U^{t}\Lambda^{\frac{1}{2}}H^{t}}}}} \\{= {\nabla{\sum\quad\nabla^{- 1}}}}\end{matrix}$

Fisher's Linear Discriminant (FLD) technique may be applied to transforma 512-dimension off-line feature into a 128-dimension feature. When thisfeature is fed into a GMM for symbol recognition purpose, dimensionreduction significantly reduces calculation costs. In addition,recognition accuracy is also improved. The FLD technique maximizes thebetween-class scatter while minimizing the within-class scatter. As aresult, the classification capacity of the reduced feature may beoptimized.

The computation of product of a high-dimensional matrix and ahigh-dimensional vector is costly. The computation cost could be reducedby various techniques, such as quantification.

Gaussian Mixture Model (GMM) is a mixture probability distributionmodel. The probability of a symbol class may be represented by a GMM. Ifthe number of symbol classes is C, C GMMs are required forclassification task. A GMM is a linear combination of K Gaussiancomponents, given by${p(x)} = {\sum\limits_{k = 1}^{K}\quad{p\left( \quad{{{x\left. k \right){p(k)}} = {{\sum\limits_{k = 1}^{K}\quad{{N_{k}\left( {{x❘\mu_{k}},\sum\limits_{k}}\quad \right)}{p(k)}}} = {\sum\limits_{k = 1}^{K}\quad{{N_{k}\left( {{x❘\mu_{k}},\sum\limits_{k}}\quad \right)}c_{k}}}}},} \right.}}$where K is the Gaussian number for each symbol, c_(k) is mixture weightsubject to constraints 0≦c_(k)≦1, and${{\sum\limits_{k = 1}^{K}\quad c_{k}} = 1},$p(x|k) is called a Gaussian component. As such,$p\left( {{{x\left. k \right)} = {{N_{k}\left( {{x❘\mu_{k}},\sum\limits_{k}}\quad \right)} = {\sum\limits_{j = 1}^{D}\quad{\frac{1}{\left( {2\pi} \right)^{D/2}\sigma_{j}^{d}}{\exp\left( {{- \frac{1}{2}}\frac{\left( {x - \mu_{j}} \right)^{2}}{\sigma_{j}^{2}}} \right)}}}}},} \right.$where D is dimension of feature (here D=128) and μ_(k),σ_(k),c_(k) aremean vector, variance vector, and priority of the k^(th) component,respectively.

Next, a set of class conditional likelihood functions is considered:g _(i)(X:Λ)=p _(i)(x),where i=1,2, . . . ,C defined by the parameter set Λ (includingμ_(k),σ_(k),c).

The classifier/recognizer operates under the following decision rule(classifier):${C(X)} = {{C_{i}\quad{if}\quad{g_{i}\left( {X;\Lambda} \right)}} = {\max\limits_{j}{g_{j}\left( {X;\Lambda} \right)}}}$

The Expectation-Maximum (EM) algorithm is a general method of findingthe maximum Likelihood Estimation (MLE). In accordance with at least oneaspect of the present invention, an EM algorithm is used to train a GMMvia EM. The following is a process of training a GMM.

The process begins with data set X of N feature vectors x_(n),n=1, . . .,N, an initial set of K Gaussian components withN_(k)=N_(k)(x|μ_(k),Σ_(k)), and K mixture weights C_(k),k=1, . . . ,K. Nis number of training symbols and the dimension feature is 128.

Then, the responsibility p(k|x_(n)) of each component PDF for eachtraining symbol feature (128-dimension) is determined as$p_{kn} = {p\left( {{{k\left. x_{n} \right)} = \frac{p\left( {x_{n}\left. k \right){p(k)}} \right.}{p\left( x_{n} \right)}},} \right.}$with GMM likelihood${p\left( x_{n} \right)} = {\sum\limits_{k = 1}^{K}\quad{p\left( {x_{n}\left. k \right){{p(k)}.}} \right.}}$

Next, components' probability distribution functions (PDFs) and weightsare re-estimated based on the data and responsibilities:${\hat{p}(k)} = {\frac{1}{N}{\sum\limits_{n = 1}^{N}\quad p_{kn}}}$$\hat{\mu} = \frac{\sum\limits_{n}\quad{p_{kn}x_{n}}}{\sum\limits_{n}\quad p_{kn}}$${{\hat{\sigma}}_{ik}^{2} = \frac{\sum\limits_{n}\quad{p_{kn}\left( {x_{in} - {\hat{\mu}}_{ik}} \right)}^{2}}{\sum\limits_{n}\quad p_{kn}}},{i = 1},2,\cdots\quad,{D.}$

The responsibility of each component PDF for each training symbolfeature is determined, and the component PDFs and weights arere-estimated until GMM likelihood${p(x)} = {\prod\limits_{n = 1}^{N}\quad{p\left( x_{n} \right)}}$of the entire training data set does not change appreciably:${{\Delta\quad L} = {\frac{L_{Current} - L_{previous}}{L_{Previous}} < {1 \times 10^{- 4}}}},$where L=log p(x).

Finally, the Gaussian Mixture Models are saved as classifier parameters.

One component in the pattern matching approach to symbol recognition isthe training algorithm, which aims to produce typical (reference)patterns or models for accurate pattern comparison. In accordance withat least one aspect of the present invention, the method of classifierdesign by way of distribution estimation and the discriminative methodof minimizing classification error rate (MCE) are used. In general,after EM training, the MCE method provides a significant reduction ofrecognition error rate. On a training sample, a loss function iscomputed to approximate the classification error, and on a trainingdataset, the empirical loss is minimized by gradient descent to optimizethe classifier parameters. Let the discriminant function of class ω_(i)equate to:${g_{i}\left( {X;\Lambda} \right)} = {\max\limits_{j}{{g_{j}\left( {X;\Lambda} \right)}.}}$

One difficulty associated with the MCE training approach lies in thederivation of an objective function that has to be not only consistentwith the performance measure, i.e., the error rate, but also suitablefor optimization. The error rate based on a finite data set is apiecewise constant function of the classifier parameter A, and thus apoor candidate for optimization by a simple numerical search method.Following the methods as described in Juang, Biing-Whang et al., Minimumclassification error rate methods for speech recognition, IEEETransaction on Speech and Audio Processing, vol. 5, no. 3, May 1997, themisclassification measure of a pattern from class ω_(i) is given by:${{d_{i}(x)} = {{- {g_{i}\left( {x;\Lambda} \right)}} + {\log\left\lbrack {\frac{1}{M - 1}{\sum\limits_{j,{j \neq 1}}\quad{\exp\left\lbrack {{g_{j}\left( {x;\Lambda} \right)}\eta} \right\rbrack}}} \right\rbrack}^{1/\eta}}},$where η is a positive number. The misclassification measure is acontinuous function of the classifier parameters Λ and attempts toemulate the decision rule. For an ith class utterance X, d_(i)(X)>0implies misclassification and d_(i)(X)<0 means correct decision. Itshould be understood by those skilled in the art that these areillustrative methods and that the present invention is not so limited tothe methods described herein.

A loss function may be defined as:${{l_{i}\left( {X;\Lambda} \right)} = {{l\left( {d_{i}(X)} \right)} = \frac{1}{1 + {\exp\left( {{- \gamma}\quad{d_{i}(X)}} \right)}}}},$where γ is normally set to ≧1. Clearly, when d_(i)(X) is much smallerthan zero, which implies correct classification, virtually no loss isincurred. Finally, for an unknown x, the classifier performance ismeasured by:${{l\left( {X;\Lambda} \right)} = {\sum\limits_{i = 1}^{C}\quad{{l_{i}\left( {X;\Lambda} \right)}1\left( {x \in C_{i}} \right)}}},$where 1(·) is the indicator function: ${1(x)} = \left\{ \begin{matrix}1 & {x\quad{is}\quad{true}} \\0 & {x\quad{is}\quad{false}}\end{matrix} \right.$

So the expected loss may be defined as:${L(\Lambda)} = {{E_{X}\left( {l\left( {X;\Lambda} \right)} \right\}} = {\sum\limits_{l = 1}^{C}\quad{\int_{x \in C_{i}}{{l_{i}\left( {x;\Lambda} \right)}{p(x)}\quad{{\mathbb{d}x}.}}}}}$

In discriminative learning, the parameters of GMM are adjusted tominimize the classification error on the training dataset. Variousminimization algorithms may be used to minimize the expected loss. Thegeneralized probabilistic descent (GDP) algorithm is an algorithm thatmay be used to accomplish this task. In the GPD—based minimizationalgorithm, the above target function L(Λ) is minimized according to aniterative procedure. The parameters are updated by following equation:Λ_(t+1)=Λ_(t)−ε_(t) ∇l(x _(t),Λ)|_(Λ=Λ) _(t) ,where t is iteration times and ε(t) is a small positive numbersatisfying:${{\sum\limits_{i = 1}^{\infty}\quad ɛ_{t}} = \infty},{{\sum\limits_{t = 1}^{\infty}\quad ɛ_{t}^{2}} < \infty},{ɛ_{t} > 0.}$

The following GMM parameter transformations maintain the followingconstraints during adaptation:${c_{jk}\left( {t + 1} \right)} = {{c_{jk}(t)} - {{ɛ(t)}\frac{\partial{l_{i}\left( {x;\Lambda} \right)}}{\partial{c_{jk}(t)}}}}$${\mu_{jkl}\left( {t + 1} \right)} = {{c_{jkl}(t)} - {{ɛ(t)}\frac{\partial{l_{i}\left( {x;\Lambda} \right)}}{\partial{\mu_{jkl}(t)}}}}$${\sigma_{jkl}\left( {t + 1} \right)} = {{\sigma_{jkl}(t)} - {{ɛ(t)}\frac{\partial{l_{i}\left( {x;\Lambda} \right)}}{\partial{\sigma_{jkl}(t)}}}}$${ɛ(t)} = {{ɛ_{0}\left( {1 - \frac{t}{T_{\max}}} \right)}.}$

In another embodiment, a two-layer classifier may be used to implementsymbol recognition. FIG. 16 is the flowchart of a two-layer symbolrecognition. First, symbol recognizer extracts off-line features 1603 ofthe strokes 1601 which may be a symbol grouped by symbol grouping.Subspace classifier 1605 is used to classify the features and givescharacter candidates with confidence. If the top-1 confidence of thefirst candidate is high enough 1607, the symbol recognizer will outputthe candidates and confidence directly 1609. Otherwise, the recognitionwill go to the second layer. On-line features will be extracted 1611 andthe nearest center classifier 1613 is used to give new charactercandidates and confidence.

The off-line classifier is a template-based classifier, which uses thesame approaches of feature extraction and dimension reduction with GMMrecognition. The online classifier is also based on a template matchingapproach. The online classifier extracts on-line features, and uses aclassic Dynamic Time Warping (DTW) algorithm to calculate the distancebetween a template and a test pattern.

For on-line feature extraction, dominant points are first detected atstrokes. Dominant points are important points that may represent strokeswell. If dominant points are removed from strokes, the strokes will bedistorted significantly. In this example as shown in FIG. 19, thefollowing three types of dominant points are used: pen-down and pen-uppoints, corner points, and mid-points. A pen-down and pen-up points areused firstly. A pen-down point is the first point of a stroke and apen-up point is the last point of a stroke. If a writing direction of apoint changes above a threshold amount from that of its previous point,we call this point as a corner point. The third type is the mid-points,which are points between two dominant points with above types. If adistance of two dominant points is too far, a third type of dominantpoint is created.

With the dominant points detected, a local feature for each dominantpoint is extracted. Local features include the tangent direction andcurvature direction. Every dominant point has a 4 dimensional feature:f_(i)=(cos α(i),sin α(i),cos β(i),sin β(i))^(T). The tangent direction(α) is shown in FIG. 17. In FIG. 20, the angle (α) between the two linesis the approximate curvature direction (β). The curvature directionfeature is described as:cos β(t)=cos α(t−1)cos α(t+1)+sin α(t−1)sin α(t+1)sin β(t)=cos α(t−1)sin α(t+1)−sin α(t−1)cos α(t+1)

Finally, the feature vector sequence: F=f₁f₂f₃ . . . f_(m), where m isthe number of dominant points, and f_(i)=(cos α(i), sin α(i), cos β(i),sin β(i))^(T) is generated.

In accordance with aspects of the present invention, symbol groupingdepends on the confidence of symbol recognition. Generally, symbolrecognition may tell the degree of similarity between a test pattern andan appointed symbol, but may not tell the degree that the given strokesare similar to a symbol.

Sometimes, strokes may not be a symbol, but symbol recognition gives avery high confidence to the top-1 candidate. As described above, FIG. 24shows two examples. In a usual symbol recognizer,

may be recognized as “H”, and

may be recognized as “F” with high confidence. In dynamic programming,

may be grouped into a symbol with high probability.

In order to resolve this ambiguity, in symbol recognition, a specialsymbol, a non-symbol, which means a stroke is not a symbol, isintroduced. In other words, the symbol recognizer views non-symbols as“special symbols”. Moreover, symbol recognition may give confidence of anon-symbol. It provides a decision base for symbol grouping. Asdescribed above, many non-symbol samples are generated from labeledexpression data. Because symbols in expression data are labeled, otherstroke combinations are all non-symbols. These samples are added into adataset as a type of “special symbol”. So if the system recognizesstrokes as the “special symbol”, these strokes are grouped into a symbolwith a lower confidence.

Symbol Grouping

The task of symbol grouping is to separate strokes into several groups,which are most likely to be symbols. If any combination of strokes isconsidered in the calculation, the searching space is very large. Forexample, the number of different ways to only separate n strokes into 2groups is 2^(n−1)−1. However, the exact group number is unknown beforesymbol grouping is done. If all stroke combination of 3 groups, . . . ,n groups is considered, the searching space is so huge that calculationfor grouping is not feasible at all.

To alleviate the computations, in accordance with one aspect of thepresent invention, an assumption is made that users write a symbol withseveral strokes which are consecutive in time order. Such an assumptionis reasonable for most handwritten expression. Although a dot for an ‘i’and a cross bar for a ‘t’ may be appended after a writer has completedan math expression, few people write half a symbol then finish thesymbol after completing the remainder of the sentence or other writing.

With this assumption, in accordance with one aspect of the presentinvention, the strokes may be sorted by time order. A consideration ismade as to how to separate a stroke sequence into several segments,where each segment is a symbol, as shown in FIG. 23. Now the number ofdifferent ways to group strokes is 2^(n−1). Although the searching spaceis still large, it has been reduced sharply. Moreover, the assumptionallows the use of a dynamic programming algorithm to search for anoptimal solution for the problem. It should be understood by thoseskilled in the art that 2^(n−1) is the count of separating strokes intoonly 2 groups if no above time order assumption. However, the group(symbol) count is unknown before the calculation. So the count may beany one of 1, 2 . . . , n, if the assumption described in the nextparagraph is not incorporated. Therefore, the number of different waysseparating n strokes without time order assumption is much greater thanthe number with time order assumption. In accordance with at least oneaspect of the present invention, the stroke count of a symbol may alwaysbe below a fixed number. Because the vast majority of mathematicalsymbols are simple, users may write them with several strokes. Such asituation also reduces the searching space. Moreover, such a situationmakes it easier to implement a dynamic programming approach after arelational score between neighboring symbols is involved.

A stroke sequence may be defined as strokes 1,2, . . . N. There may bemany different ways to separate strokes in the stroke sequence intosegments, e.g., to group the strokes into groups. For example, (n₁=1,n₂−1), (n₂,n₃−1), . . . (n_(k−1),n_(k)−1=N) is one illustrative way toseparate the N strokes into segments. A measurement may be defined tomeasure the different ways of grouping. With a defined measurement,different ways of grouping strokes may be compared, and an optimizationtechnique may be used to find the solution.

In accordance with at least one aspect of the present invention, a wayto group strokes may be evaluated based upon two aspects. First, adetermination is made as to how likely a group is a symbol according tointra-group information. For the given segmentation, there are p(1), . .. , p(k) scores to measure the k groups. Second, a determination is madeas to the relationships between neighboring groups to determine howlikely two groups may be neighbors. p(i|i−1) is defined to be therelational score between the i−1 ^(th) and i^(th) groups, where i isfrom 1 to k. If i is equal to 1, p(1|0) is the likelihood that the groupwould be a first symbol of the stroke sequence. Therefore, themeasurement for a way to group strokes may be defined by:${Measurement} = {\prod\limits_{i = 1}^{k}\quad\left( {p\left( {i\left. {i - 1} \right) \times {p(i)}} \right)} \right.}$

Although the searching space is reduced sharply, it is practicallyinfeasible to calculate the scores for all ways and compare them to getthe optimal one by brute force. There is a good characteristic in thegrouping problem. If s₁, . . . , s_(k) are symbols for a strokesequence. If the strokes of s_(k) are erased, s₁, . . . , s_(k−1) arethe symbols for the remaining stroke sequence. Formulaically, if (n₁=1,n₂−1), (n₂,n₃−1), . . . (n_(k−1), n_(k)−1=N) is an optimal way ofgrouping the N strokes, then (n₁=1,n₂−1), . . . (n_(j−1),n_(j)−1) isalso an optimal way of grouping the sub-sequence (n_(i−1),n_(j)−1) ofstrokes. As such, dynamic programming may be utilized to obtain a globaloptimization based on the combinations of the local optimizations.

S(1, i) is defined as the score of the optimal segmentation for sequence(1, i). If the scores S(1, i), where i is from 1 to N−1, are alreadyknown, the optimization for sequence (1, N) may be calculated as:${S\left( {1,N} \right)} = {\underset{l \leq L \leq 5}{Max}\left( {{{S\left( {1,{N - L}} \right)} \times {p\left( {{N - L + 1},{N\left. {1,{N - L}} \right) \times {p\left( {{N - L + 1},N} \right)}}} \right)}},} \right.}$where L is the length of last group, S(1,N−L) is optimal score ofsequence (1, N−L), p(N−L+1,N|1,N−L) is the likelihood that group (N−L+1,N) may be the next group of the optimal groups of sequence (1, N−L), andp(N−L+1N) is the likelihood that (N−L+1, N) is a symbol. There,p(N−L+1,N|1,N−L) only depends on the last group of sequence (1, N−L).

The use of space analysis alone to determine whether several strokes maybe a math symbol is rather difficult. FIG. 22 shows an example of thisproblem, where strokes within box 2203 are not much different fromstrokes within box 2201 from the perspective of spatial information.However, strokes within box 2203 should be grouped to a symbol becausethe strokes are part of the character ‘A.’ Therefore, besides spatialinformation, symbol recognition is an important information source forsymbol grouping. Symbol recognition may output possible charactercandidates and their corresponding confidences for any given strokes,which may be utilized in the calculation for grouping.

Usually, symbol recognition assumes given strokes must be a predefinedsymbol, and symbol recognition outputs the top-n likely candidates forthe given strokes. For the task of symbol grouping, one aspect is thatsymbol recognition determine if given strokes are a predefined mathsymbol, and with how much confidence. The non-symbol is similar to acommon symbol, so that symbol recognition may recognize the non-symboland give its confidence. But the non-symbol is also a special symbol,which means given strokes are not a math symbol if the strokes arerecognized as a non-symbol.

Thus, symbol recognition may output the top-n character candidates andtheir confidences. Moreover, there may be possible non-symbols in thecandidates, with the summation of all confidences being equal to 1. Asdescribed above, determination of a score, which is likelihood thatgiven strokes are a math symbol would be helpful. In accordance with atleast one aspect of the present invention, the score S_(r) is defined asthe confidence summation of all candidates which are symbols.

Usually, system recognition has to normalize inputting strokes to itsinner scale, and the recognition operates best in the inner scale. Butdifferent from text handwriting, where characters are almost the samesize, the sizes of math symbols may vary, especially when themathematical symbols are located at different levels of within anexpression. Under such conditions, the normalization loses some neededspace information. Several spatial features may be used to compensatefor this weakness.

As described above, the distance feature (d) is one feature fordetermining grouping. This feature prevents over-grouping of strokes. Atdifferent levels of grouping such a symbol, the function string, thedefinition and the calculation of distance are different. In (a) of FIG.21, there is an obvious distance between ‘a’ and ‘+’, so ‘a’ and ‘+’should not be grouped together.

The size difference feature (δ) prevents a symbol and its subscript orsuperscript from being grouped. In (b) of FIG. 21, the size differencemay be used to distinguish some special letters, such as lower case ‘o’and upper case ‘O’. Without this feature, it may be difficult for arecognition engine to distinguish lower case and upper case letterscorrectly.

The offset feature (σ) is another feature in symbol grouping which isused to guarantee symbol strokes located in the same horizontal line. Asshown in (c) of FIG. 21, “a+b” and “c+d” are not grouped into a symbolbecause there is a fraction line between them.

After the feature extraction, a decision function is constructed tocombine the features to get a score, which is the probability that thestroke set is a correct group. The decision function is constructed asthe following:${{f\left( {d,\delta,\sigma} \right)} = \frac{1}{1 + \left( {\left( \frac{\mathbb{d}}{\mathbb{d}_{0}} \right)^{\alpha} + \left( \frac{\delta}{\delta_{0}} \right)^{\beta} + \left( \frac{\sigma}{\sigma_{0}} \right)^{\gamma}} \right)}},\left( {\alpha,\beta,{\gamma > 1}} \right),$where d₀,δ₀,σ₀ are the thresholds of the three geometrical features. Theα,β,γ are similar to the λ of a sigmoid function and are used to tunethe softness of the decision boundary. With the function, a score S_(s)based on spatial information, which also measures how likely a strokeset is a symbol, may be determined.

Now, with the two scores from intra-group information, the followingformula may be used to combine the two scores to determine a finalintra-group score:S _(w)(S _(r))^(p)×(S _(s))^(1−p),where p is a weight corresponding to how much the score given by thesymbol recognition subsystem may be trusted. Moreover, a differentweight p may be used for a different character. For example, if thesymbol recognition subsystem gives robust confidence for character ‘A,’a big weight p for character ‘A’ may be used. By doing this, betterflexibility in determining symbol recognition may be obtained.

Besides weight p, d₀, δ₀, σ₀ and α,β,γ also depend on the charactergiven by symbol recognition. In accordance with at least one aspect ofthe present invention, supported characters are categorized into certainnumber of clusters. Each cluster has independent model parameters, whichmay be trained with a training program. The design of cluster-dependentparameters achieves better accuracy performance than a system with onlyone set of parameters.

A mathematical expression is a two dimensional (2D) layout of symbols.Some symbols have unique spatial structures. For instance, symbol ‘Σ’usually has other associated symbols in the regions directly above anddirectly below the symbol, such as shown in FIG. 34. The rich spatialinformation may be used to solve the grouping problem. A typical exampleis the symbol ‘θ’. If inter-group spatial information is not considered,symbol ‘θ’ is often separated into two symbols: ‘0’ and ‘−’, such asshown in FIG. 28. However, it is not possible that ‘0’ and ‘−’ overlapeach other spatially in an expression. This correlates to theunderstanding that the separation is incorrect.

In the system in accordance with at least one aspect of the presentinvention, nine spatial relations are defined for inter-symbol spatialrelationships, such as shown in FIG. 27. They are horizon, superscript,subscript, above region, below region, overlap, left-horizon,left-super, and left-sub. During the calculation for grouping problem,whether a break is correct or not is not know. If a tentative break liesin a symbol, the relationship of two groups separated by the break areclassified into nine classes. However, such a configuration of the nineclasses is meaningful only for inter-symbol spatial relations. As such,in accordance with at least one aspect of the present invention, theintra-symbol relationship is defined to better model the situation of abreak lying in a symbol, such as shown in FIG. 28.

In sum, there are ten spatial relationships defined in the system. Giventwo neighboring groups, spatial features are extracted from the twogroups. A Gaussian Mixture Model may be used to fit its featuredistribution for each relationship. With the Gaussian models, aclassifier identifies the spatial relationship between two groups.Moreover, the classifier may give the confidence for the identifiedrelationship. The confidence may be defined as p(R|F), where F is thespatial features and R is one of ten relationships.

A mathematical expression is also a syntax structure. Although a usermay write math symbols in any order, which perhaps is not consistentwith the syntax structure, it is still reasonable to utilize thetemporal context information to calculate grouping. For example, if auser writes a digit, it is possible that the next written symbol is alsoa digit. Therefore, a bi-gram probability may be built in the system toutilize the temporal context information.

The bi-gram probability is built by combing spatial relationshipstogether. The bi-gram probability is defined as p(S₂|S₁,R), where S₁ isthe previous character, S₂ is the next character, and R is one of tenrelations. p(S₂|S₁,R) may be calculated by:${{p\left( {{S_{2}❘S_{1}},R} \right)} = {\frac{p\left( {S_{1},S_{2},R} \right)}{p\left( {S_{1},R} \right)} = \frac{C\left( {S_{1},S_{2},R} \right)}{C\left( {S_{1},R} \right)}}},$where C(S₁,S₂,R) represents the count of events of a previous characteris S₁, and a next character S₂, and their relationship is R. C(S₁,R)represents the count of events of a previous character S₁ and Rrepresents the relationship with a next symbol.

Because the bi-gram probability p(S₂|S₁,R) depends on the characters ofthe symbols, given two neighboring groups, symbol recognition may outputtheir character candidates and confidences. An inter-group score may becalculated as:$S_{b} = {\prod\limits_{S_{1},S_{2},R}\quad\left( {{p\left( S_{1} \right)} \times {p\left( S_{2} \right)} \times {P\left( {R❘F} \right)} \times {P\left( {{S_{2}❘S_{1}},R} \right)}} \right)}$Tabular Structure Analysis

Returning to FIG. 3, tabular structure analysis for handwrittenmathematical expressions component 305 is another module in themathematical expression recognition system. Tabular structure includesmatrix and multi-line expression. It may be useful to divide a group ofhandwritten strokes into columns and rows and thus form matrices ormulti-line expressions. Each cell of the results may be furtherrecognized as a sub-expression by other modules or recursively processedif it still contains tabular structures.

The algorithm of tabular structure analysis in the recognition systemfor mathematical expression includes three parts. Firstly, X-Yprojection divides the inputted strokes into rows and columns. Thisaffects those divisible parts and has no negative effect for non-tabularstructures. Secondly, those candidates of tabular structures given inthe previous step are accepted or rejected by judging whether bracketsexist. Thirdly, some rows and columns may be merged to correct the oversegmentation problem in the X-Y projection. The main difference betweenmatrices and multi-line expressions is the surrounding brackets. This isjudged in the second step. If a structure is judged to be a multi-lineexpression, all columns may be merged into one column in the third step.

Tabular structure analysis is the process of dividing strokes into rowsand columns. Blank is a feature for tabular structures. Rows and/orcolumns are divisible when there are blanks among them. An X-Yprojection is used to identify the blanks in the rows and/or columns ofstrokes, such as shown in FIG. 31.

The following is an illustrative implementation with respect to FIGS. 31and 32. Firstly, a whole input block is projected on the X axis anddivided into columns. For example, as shown in FIG. 31, the whole inputblock is divided into seven columns, C₀, C₁ . . . C₆. Secondly, themaximum divisible columns from left to right are searched incrementally.Each column is projected to the Y axis with previous divisible columnsto judge whether these columns are also divisible. If divisible,searching continues to the next divisible column. Otherwise, adetermination is made as to whether previous divisible columns exist. Ifprevious divisible columns do not exist, no further determinations areneeded and the process continues. Otherwise, the previous divisiblecolumns are used to form a tabular structure and previous divisiblecolumns are set to empty.

For example, column C₀ in FIG. 31 is first processed and shown asindivisible. Moving to the next column, column C₁ is divided into 3rows. Column C₂ is the same as column C₁ and the combination of thesetwo columns also form 3 rows. Although column C₃ has only one stroke, asshown in row R₂, and thus is indivisible, the Y projection on thecombination of columns C₁, C₂, and C₃ identifies a structure of 3 rowsand 3 columns. After processing columns C₄ and C₅, a tabular structureof 3 rows and 5 columns is obtained. Because over segmentation problemexists (e.g., columns C₃ and C₄ should not be separated from column C₂),such a tabular structure may contain empty cells which will beeliminated later. Column C₆ is then processed next. As with column C₀,column C₆ is indivisible so the previous five columns are regarded as acandidate of a tabular structure and processing continues as describedin the following.

A bracket is a component of tabular structures. If a pair of brackets isfound at the left and right sides of a candidate which is given in theprevious X-Y projection, the candidate is accepted as a matrix. If thereis only one curved bracket at the left side and the right side has anopening, the candidate is accepted as a multi-line expression.Otherwise, the candidate is rejected and processed by other modules asappropriate. For example, if some superscript and subscript elements arealigned as a vector, they may be a candidate given by X-Y projection.However, because the left and right strokes are not brackets, the falsecandidate may be rejected at this point.

A symbol should satisfy the following two requirements to be accepted asa bracket. One is that its symbol recognition result should be a validbracket which is described above. The other requirement is that theheight of the symbol is large enough to encapsulate the candidate andthat the proportion between the width and the height of the symbol issmall enough in comparison to one or more thresholds. Such criterion maybe controlled by two pre-defined thresholds.

A simple X-Y projection may introduce the over segmentation problemdescribed above with reference to FIG. 31. The system may not beconfigured to support matrices which contain empty cells. If somecolumns which have empty cells are found after the X-Y projection, theymay be merged with one of the neighboring columns. The distance to theleft neighbor and right neighbor may be compared and such a column willbe merged with the nearer neighbor. For example, columns C₃ and C₄ inFIG. 32 are nearer to column C₂ comparing with the distance to columnC₅. So columns C₂, C₃, and C₄ are merged into one.

Next, the distances between any two neighboring columns are comparedwith a pre-defined threshold. Those columns whose distances are veryshort may also be merged to reduce over segmentation further. If theright side of the tabular structure is opening, then it is recognized asa multi-line expression and all columns are merged into one. Finally,rows may be also merged when the distances between rows are short enoughwhen compared to a pre-defined threshold. Such a determination andprocess may be used to correct the over segmentation for those cellsthat only contain a fraction.

Subordinate Sub-expression Analysis

Mathematical expression is data with structure information. Besidessubscript and superscript structures, there are above and below spatialrelations (Σ) and pre-superscript spatial relations (√) in expressions.To represent the relationship between symbols, people use more layouttypes for expressions in both handwritten notes and printed documents.This makes expression structure analysis much different than text layoutanalysis, where regular words, lines, paragraphs exist.

The data structure of a mathematical expression is inherently a treestructure. Logically, an expression may be divided into severalsub-expressions and a sub-expression may be subordinate to a symbolwhich is in another sub-expression. With the subordinate relationship,the sub-expressions form a tree. Therefore, the inherent hierarchicalsub-expression structure for the system may be found. For furtherprocessing, an entire expression may be subdivided into several parts.The following describes the task as structure analysis.

In accordance with at least one aspect of the present invention, twosub-expression types are distinguished to be handled. The first type isa sub-expression subordinate to special structural symbols, such as Σ,√, ∫, which are named as dominant symbols. These symbols always implyunique layout relations existing in expressions. The second type is asub-expression of a subscript or a superscript, which also often appearsin common text. In accordance with one aspect of the present invention,the first type of sub-expressions is found in an expression and therecursive structure is then determined. The other type will be processedby the next component.

The subordinate sub-expression analysis component is a component of thesystem of expression recognition. Three points are described again forthe sake of consistency. A parse tree is used and passed by allcomponents in the mathematical expression system. The parse tree may bean extended BST tree, which is defined herein.

FIG. 3 shows the framework of the recognition system. The subordinatesub-expression analysis component 307 gets symbols from the parse tree,analyzes these symbols, and writes the multi-results back to parse tree.Arriving at this point, the parse tree includes multiple results of theprevious symbol grouping and recognition component 303. This componentcontinues to handle all these results respectively.

As described herein, there are so many ambiguities in a handwrittenmathematical expression. Structure ambiguity is one of the ambiguities.Sometimes, it is not easy to judge if a symbol is inside a radical signor not. FIG. 33 is such an example. The ambiguity is whether the symbol“c” is inside or outside the radical sign. In accordance with at leastone aspect of the present invention, multiple results are outputted toresolve these kinds of ambiguities.

As mentioned herein, an expression is a tree of sub-expressions. FIG. 33shows this concept clearly. In FIG. 33, the content in each rectangle isa sub-expression. The left two sub-expressions are subordinate to thefraction line and the right sub-expression is subordinate to the radicalsign. The two dominant symbols lie in the main sub-expression. The foursub-expressions form a tree structure. Sub-expressions may include oneor more symbols. Within a sub-expression, there are no other spatialrelations except horizontal spatial relations between symbols. Thiscomponent mainly analyzes the first type of sub-expression, namely thesubordinate sub-expression. In the component, subscript and superscriptare handled in the same way to deal with horizontal relationships. Thesubtle distinctions among them are processed by the next component.

Dominant symbols imply particular layout types in expressions. They areseparated from other symbols and used as hints by this component. InTable-1, the rows are dominant symbols supported by the component so farand the columns are the types of their relationships with thecorresponding sub-expressions. The marks in cells of the table body meandominant symbols may have the corresponding types of sub-expressions.For example, there are two cells marked in the first row, so thatfraction line may have two sub-expressions, one is the numerator abovethe fraction line and the other is the denominator below the fractionline. TABLE 1 Example Dominant Symbols and Relationships Be- Above lowContain Index — (fraction line) ✓ ✓ Π (N-Array product), ✓ ✓ Σ (N-Arraysum) ∫ (Single integral sign) ✓ ✓

, ∫∫, ∫∫∫^((Other integral signs)) ✓ √ (radical sign) ✓ ✓ -, → (hatsymbols) ✓

This component uses a graph search algorithm, which includes the stepsof constructing a relational graph and searching the Top-N optimizedspanning tree. In the graph, vertexes are symbols and edges are possiblerelationships between symbols and their corresponding intensity. It isalso possible that there are multiple relations between two symbols dueto spatial ambiguities.

The graph is not the final description of symbol relationships. Thereare many conflicts in the graph. One is, as mentioned above, multiplerelationships exist between two symbols, but actually only one is valid.Another example is a symbol may be subordinate to multiple symbols inthe graph. So after graph construction, a search process is performed inthe graph to decide which relations are valid. These valid relations(edges) finally form an optimal spanning tree on the graph. Moreover,the search algorithm investigates almost all possible combinations ofedges during the process. It may evaluate all combinations, which arespanning trees, and record Top-N optimal results. This componentperforms the following two tasks. First, the component finds subordinatesub-expressions for each dominant symbol. By doing this, Top-Nhierarchical trees of a sub-expression are constructed. These multipleresults are mapped to a parse tree for further processing.

Second, the component decides characters of dominant symbols. The symbolrecognition component supplies a list of character candidates for eachsymbol, but the character of the final symbol is still undetermined.Actually, it is impossible to decide a unique character for each symbolonly by symbol recognition. For example, ‘Minus’ and ‘Fraction line’ maynot be classified from each other by a symbol recognizer. For such acase, structure context information is needed, because ‘Fraction line’has two sub-expressions—denominator and numerator. So the component alsomay determine characters of dominant symbols with structure information.

The input of this component is a handle of the parse tree. By thishandle, this component may access the whole parse tree. Arriving at thispoint, the parse tree has been processed by the symbol grouping andrecognition component. It has created some symbols grouping andrecognition solution nodes in the parse tree to represent the multipleresults of the symbol grouping and recognition component. For anintuitive image, an example snap shot of the parse tree at this time isgiven at FIG. 48. This component accesses one solution node, gets allthe descendent symbols node of the solution nodes, processes the symbolsnodes, writes back multi-results, accesses another solution node andcontinues until all symbols grouping and recognition solution nodes areprocessed. This component will create a new subordinate sub-expressionanalysis solution node in the parse tree for each result. Processed bythis component, the parse tree may look like FIG. 49.

In construction of a graph, calculating relational scores for edges maybe needed. A relational score is a measure of the intensity of arelationship. Five relational types are taken into consideration. Besidethe four relational types in Table 1, the horizontal relationshipenabled for any math symbol is considered. So for each couple of mathsymbols, there are five possible edges between them initially. Edgeswith a lower score than a specified threshold are removed in order toreduce memory cost and time cost. The following are some concepts in thecalculation.

For each symbol and for each enabled relational type, a rectanglecentered control region is calculated from a fairly large training set.The control region is a centered rectangle which is infinite andtruncated. In FIG. 34, the two shadowed rectangles represent the tworectangle centered control regions for ‘Above’ and ‘Below’sub-expression types respectively. FIG. 37 is an example to describe howthe control region is truncated.

The score is calculated to measure to what extent a point (x, y) issubordinate to a specified control region according to sub-expressiontype R. If the point is located inside the centered rectangle of acontrol region, the score will be set to 1.0, the biggest score value.In the alternative, if the point is not located in the control region,the score will be set to 0.0, the smallest score value. The generalprinciple when calculating a relational score is that the nearer thepoint is to the centered rectangle, the bigger the score will be and thefarther the point is to the centered rectangle, the smaller the scorewill be. FIG. 35 is the equation used to calculate the score. In FIG.35, f_(R)(x,y) represents the score. O_(R)(x),O_(R)(x) represent theoffsets of the point (x, y) to the according rectangle respectively.λ_(x),λ_(y),x₀,y₀ are specified thresholds. FIG. 36 is the graphicaldescription of the equation in FIG. 35.

Given a symbol, the bounding box may be determined. This componentcalculates symbol score to a control region by the correspondingbounding box. First, it samples a specified large number of points inthe bounding box uniformly. Second, it calculates point relational scorefor each sampled point one by one using the method mentioned above.Third, it averages all those score obtained at the second step to getthe symbol relational score. FIG. 38 is a formal description where S isthe bounding box of a symbol to calculate relational score, R is theaccording infinite but truncated control region and (x, y) is point inS. FIG. 39 is an intuitive description of such an operation.

The relational score from the previous step has a shortcoming in that itdoes not take the global information into consideration. But a thirdsymbol may affect the relationship between two symbols. There are twocases. The first case is that the subordinate symbols subordinates to amore specific dominant symbol. For example, in the left part of FIG. 40,symbol “a” is above the fraction line and is contained by the radicalsign. Because the radical sign is above the fraction line, it is themore specific dominant symbol related to the symbol “a”. In this case,the radical sign affects the relationship between the symbol “a” and thefraction line. There is no direct relationship between the symbol “a”and the fraction line at all because of the existence of the morespecified radical sign. The other case is that two symbols having somerelationship with each other are separated by a third dominant symbol.The right part of FIG. 40 shows this case. If the fraction line does notexit, the index relationship between the symbol “3” and the radical signwill be assigned a high score. Such does not occur here. Because of theexistence of the fraction line, the symbol “3” becomes the numerator andthe radical sign becomes the denominator. The two symbols have no directrelationship any more.

The relational score needs to be adjusted with reference to globalinformation for both of the two cases mentioned above. For the firstcase that the subordinate symbols subordinates to a more specificdominant symbol, the original relational score is subtracted a valueequal to the product of two relation scores. One is the relational scorebetween the subordinate symbol and the more specific dominant symbol.The other is the relational score between the more specific dominantsymbol and the subordinate symbol. In FIG. 41, the above relationalscore between the symbol “a” and the fraction line is subtracted fromthe value of the relational score between the symbol “a” and the radicalsign which is a more specific dominant symbol to the symbol “a”. For theother case that two symbols having some relationship are separated by athird symbol, the relational score between the two symbols is subtractedfrom a value equal to the product of the relational scores between thethird symbol and the two symbol respectively. In FIG. 42, the indexrelational score between the symbol “3” and the radical sign issubtracted from a value of the product of the above relational scorebetween the symbol “3” and the fraction line and the below relationalscore between the radical sign and the fraction line which is aseparator. Generally speaking, to adjust the relational score betweentwo symbols, all the other dominant symbols must be gone though toperform the two rules mentioned above. FIG. 43 is an overall formalequation to be used to adjust the relational score by global informationin this component.

In this graph, an edge represents a relationship between two linked mathsymbols. Because of the ambiguity in handwritten mathematicalexpressions, there may be more than one relationship between each coupleof math symbols. An edge will be created for each couple of symbols andfor each relational type. In order to build such a graph, the horizontalrelationship is also taken into consideration. All math symbolsincluding dominant symbols may have a horizontal relationship with thesymbols behind them. So there will be two types of edges namely thepaternity edges and the brotherhood edges in the obtained graph. Inorder to reduce time cost and storage cost, edges with relational scorelower than a specified threshold will be pruned. FIG. 44 is such arelational graph.

The recursive structure of a mathematical expression may be expressed bya tree. So a search process will be performed in the relational graphfor the according tree structures. The search process considers both thetwo types of edges for each symbol. In order to resolve the structureambiguities of mathematical expression, the top-N optimized spanningtree will be reserved. FIG. 45 is the search process and FIG. 46 is theinput and output of the search process.

Results found by the previous search process are optimal only withrespect to local relational scores, and thus may not guarantee that theresult is valid globally. For example, in a result found by the searchprocess, the fraction line may have a numerator, but no denominator. Butthis kind of global structural information must be considered. Sostructure validity checking is involved in the subordinatesub-expression analysis component to verify if the results are valid,after previously finding multiple results. Finally, only valid andoptimal results will be outputted to the parse tree. An overallconfiguration of the subordinate sub-expression analysis component isillustrated in FIG. 47.

Subscript, Superscript Analysis and Character Determination

The symbol grouping and recognition component supplies multiplecharacter candidates with confidences for each symbol. The subordinatesub-expression analysis component finds out sub-expressions for eachdominant symbol but it does not step into subtle distinctions amongsubscript, super script and horizontal relations within eachsub-expression. This component performs two tasks, one is to select aunique character for each symbol and the other is to analyze thesubscript and superscript structures within a sub-expression. In orderto deal with the ambiguities existing in a handwritten mathematicalexpression, aspects of the present invention adopt a graphical searchalgorithm. The first step is to build a graph for a sub-expression andthe second step is to search in the graph for the top-N optimizedspanning trees each of which represents a unique mathematicalsub-expression.

The subscript, superscript analysis and character determinationcomponent is a component of the whole handwritten mathematicalexpression recognition system which aims to supply a natural way forhumans to input a mathematical expression into computers. The input ofthis component is a handle of a parse tree. By this handle, thecomponent may access the whole parse tree. Arriving at this point, theparse tree has been processed by the subordinate sub-expression analysiscomponent. It analyzes sub-expressions associated with dominant symbolsand creates a new relational node for each such sub-expression. Dominantsymbols also belong to some sub-expressions. For an intuitive image, anexample snap shot of the parse tree at this time is given at FIG. 59.This component accesses one relational node, gets all the child symbolnodes of the node, processes the symbols nodes, writes backmulti-results, accesses another relational node and continues until allthe relational nodes are processed. This component will create a newsubscript/superscript and character determination solution node in theparse tree for each result. Processed by this component, the parse treemay look like FIG. 58.

The algorithm deals with each sub-expression in the same way. FIG. 50 isthe flowchart of this algorithm. The first step is to sort all thesymbols in a sub-expression from left to right. After the symbols aresorted, a graph is built based on the symbols. Then a searching processis performed on the graph to find the top-N optimized spanning trees.Each spanning tree represents a unique mathematical sub-expression. Toconfirm the validation of each spanning tree, a syntax analyze processis performed on each spanning tree. In this process, invalid spanningtrees are removed. The last step is to write back the multiple possibleresults of the sub-expression to the parse tree. Not only the structureof the sub-expression is unique, but also the character of each symbolis also unique in a specified spanning tree.

There are only three types of relationships namely subscript relation,horizontal relation, and superscript relationship within asub-expression. A graph includes vertexes and edges. Each vertexrepresents a particular symbol of the given sub-expression. For eachcouple of symbols, for each character of a symbol and for eachrelational type being considered an edge will be created. The edgerecords the characters of two linked symbols and the relational typebetween them. In addition, a score as an intensity measure of an edge isalso recorded in the edge. In order to reduce the storage cost and timecost, any edge with a lower score than a specified threshold is pruned.FIG. 51 is a finished graph. The score of an edge is the product ofthree parts as shown in FIG. 53. The first part is a space score whichrepresents the spatial relationship between the two linked symbols. Thesecond part is a context probability score, such as a bi-gramprobability, which represents a short syntax grammar for mathematicalexpressions. The third part is the product of confidences of thecorresponding two symbol characters which come from the symbol groupingand recognition component. FIG. 52 is the equation to calculate the edgescore. In this equation, A, B are two symbols and R is a specifiedrelational type. The left part is the score of R relationship between Aand B. There are four factors in the right part. The first one is thenormalized space score for the R relationship between A and B. Thesecond part is the context probability, such as a bi-gram probability.The last two factors are confidences of A and B respectively supplied bythe symbol grouping and recognition component.

In order to calculate the space score for symbols A and B with respectto relation R, an offset in a vertical direction is calculated by theequation in FIG. 54 firstly. The second step is to calculate space scoreby the equation in FIG. 55, and then normalize the space score.

Given the characters of A and B and the specified relational type, abi-gram probability may be expressed by the equation in FIG. 56. It is aconditional probability of B, given the characters of A and the relationR. A large mathematical expression set is used to obtain bi-gramprobabilities for all couples of characters and for all the threerelational type in consideration. This information is kept down in atable. So, the bi-gram probability for two symbols with respect to aspecified relational type may be looked up in a prepared table.

The task of the next step is to find the top-N optimized spanning treesfrom the built graph that is to select the n−1 best edges from all theedges in the graph if there are n vertexes under the followingconstraints. Edges in the spanning tree must agree with each other inthe structure of the mathematical expression. Edges in the spanning treemust agree with each other in the character of each math symbol.

The search process gets more than one spanning trees each of whichrepresents a unique mathematical sub-expression. Because the searchprocess only utilizes local information, the obtained top-N spanningtrees may not represent valid mathematical expressions. In order toresolve this problem, each spanning tree will be analyzed by theinherent grammar in mathematical expressions. The well known Earley'scontext-free parsing algorithm, as described in Grune, D. and Jacob, C.J. H., Parsing Techniques: a practical guide, Ellis Horwood, Chichester,1990 and Earley, J., An Efficient Context-Free Parsing Algorithm, Comm.ACM 13, 2 pp. 94-102, February 1970, is adopted here. It should beunderstood by those skilled in the art that the above describedalgorithm is commonly known and understood by those skilled in the art.Spanning trees are converted to linear format that may be analyzed bythe algorithm. Only valid spanning tree may pass the algorithm. Thosespanning tree that cannot pass the algorithm will be removed. FIG. 57gives an example of this process.

Mathematical Expression User Interface

Whether due to illegible or poor handwriting of a user or an incorrectevaluation of strokes, inaccurate results may occur. In response, theuser will need to correct the inaccuracies. In accordance with oneembodiment, there are two places where a correction user interface maybe provided: on ink, the handwritten version, or on text, the recognizedversion. Once ink is recognized, structures in ink are identified. Inkstrokes are grouped into symbols and sub-expressions are identified.Corrections on ink may be provided based on the ink structures such assymbols and sub-expressions.

A user interface (UI) in accordance with aspects of the presentinvention allows users to modify recognized results and helps users toget mathematical expressions correctly, easily, and efficiently. Inaccordance with aspects of the present invention, the UI may be an inputpanel, a dialog, or other type of UI that allows a user to handwrite,convert, and/or correct the recognition results and to place the resultsinto an application program the user wants placed, such as into a wordprocessing application program document. An example UI of a mathematicalexpression input panel may be divided into four parts: an input orhandwriting area, a rendering or display result area, a tools area, anda function panel as shown in FIG. 60.

One part of the interface is the input area or handwriting area. Usersmay write, erase, and select strokes in the input area. There are threemodes for the input area: writing, erasing and selecting. The modes areindicated by three icons in the mode area at the left of the input area.The recognition results are shown in the rendering area or resultdisplay area below the input area. The area may be shown automaticallyafter the program gets the parse result and hidden automatically whenusers begin to write or erase, or the area may be shown at all times.The description text may be also shown in the rendering area after usersclick on the icons in the function panel. There is a button “Insert” atthe right of the rendering area. After getting the desired result, usersmay click “Insert” to send the results to the active application. Thefunction panel is at the right of the input area. There are twelveicons, which represent different function names, in the function panel.The whole layout is compact and functional.

There are two types of ambiguities in the results of mathematicalrecognition, structural ambiguity and symbol ambiguity. For example, theoriginal strokes have two different grouping schemes in FIG. 61 and eachscheme is reasonable. The stroke

may be interpreted as ‘c’, ‘(’ or ‘l’ and each interpretation may becorrect under different conditions. Candidates for a symbol,sub-expression and the entire expression are provided by the underlyingmathematical expression recognition engine. Providing candidates makesit easy for users to make a choice to correct the recognition errors.

A thin line is displayed underneath each sub-expression and the entireexpression to indicate that there are candidates. When users hover on aline, the line becomes thickened, as shown in FIG. 62. Users may thenclick on the thickened line. After clicking on the line, a candidatemenu will pop up, as shown in FIG. 63. If a pen hovers above thebounding box of one symbol, the color of the symbol will be changed, forexample to gray, indicating there are candidates for the symbol. After auser clicks on the symbol (actually, anywhere inside the bounding box ofthe symbol), the candidates menu will pop up for users to make a choice,as shown in FIG. 64. Another way users may open the candidates menu isto click anywhere inside the bounding box of a symbol, a sub-expression,or the entire expression, and the system will open the candidates menufor the unit that has the smallest bounding box encompassing the spotthe user clicked on. This method allows users not to have to accuratelyposition the pen. Users may click and select in a large enough area toget the candidates menu. The application minimizes the number ofoperations of the user.

To provide better candidates, candidates at different levels may beprovided. For example, when the whole expression is selected, the firsttime users click to open the candidates menu, top n candidates, where nmay be any reasonable number, for example, 3, 5, 8 and so on, of thewhole expression may be shown, as shown in FIG. 63. After users choose acandidate from the candidates menu, e.g., the first candidate, morecandidates with the same grouping scheme as the candidate selected inthe first round may be shown, as shown in FIG. 65. This gives users morechoices. Typically, the more candidates shown, the more likely thecorrect recognition result may be in the candidates list. This maybemeasured by the accuracy of symbol recognition and structure recognitiongiven the number of candidates. For example, in one implementation, theaccuracy of symbol recognition increases by about 6.5% when the numberof candidates provided is increased from 1 to 5. Similarly, the accuracyof structure recognition increased about 8% when the number ofcandidates provided is increased from 1 to 5. Because candidates areprovided for symbols, sub-expressions and the entire expression, thechances that the correct symbol, sub-expression and whole expression isprovided are increased.

The mathematical expression input panel is a pen-based application andthe interactions may be optimized for a pen. For example, in oneimplementation, the program may launch the parser automatically twoseconds after the user stops editing. The rendering area may be shownautomatically after the program gets the parse result and hiddenautomatically when the user begins to write or erase. A symbol erasermay be implemented, e.g., after the recognition, when a user uses theeraser to erase all strokes of a symbol at a time. The reason for thesymbol eraser is when a user erases some strokes after the recognition,it is more likely there is an error with the whole symbol. With thesymbol eraser, a user may erase more than one stroke in one removaloperation.

There are three dashed lines in gray as reference lines in the inputarea, as shown in FIG. 66. The middle line gives a user a referencebaseline to write. The top and bottom lines give the user upper andlower reference limit lines respectively. The three lines maybe designedsuch that they do not interfere with the user's focus on the strokes.

Due to the algorithm limitation, there may not be a right choice undersome conditions, as shown in FIG. 67. Users may specify the meaning ofrelated strokes in the mathematical expression input panel through someoperations. There are twelve functions in the function panel, as shownin FIG. 60. From top to bottom, from left to right, they are “Regroup,”“Promote,” e.g., to a superscript, “Demote,” e.g., to a subscript,“Radical Expression,” “Fraction,” “Integration,” “Summation,” “Product,”“Function Name,” “Parenthesis,” “Square Bracket,” and “Curly Bracket.”There are ToolTips attached to all icons in the function panel. Theeffect of each function is represented by their names. For example, thefunction “Radical Expression” specifies the selected strokes as aradical expression.

The functions may be divided into three parts according to theiroperations. The operations of “Radical Expression,” “Fraction,”“Integration,” “Summation,” and “Product” are:

Select some strokes, as shown in FIG. 68.

Click the icon in the function panel. In this example, the icon is

Act according to the description text in the rendering area to specifythe power and root of the evolution. In the example, since there is nopower, users will click button “Cancel”, as shown in FIG. 69. Then userswill select the root of the evolution, as shown in FIG. 70. Afterpressing button “OK”, the correct result is shown in FIG. 71.

The operations of “Regroup”, “Promote”, “Demote” and “Function Name”are:

Select some strokes.

Click the icon in the function panel.

The strokes selected will be grouped together and recognized as a singlesymbol or promoted, such as becoming a superscript, or demoted, such asbecoming a subscript, or recognized as a function name, such as sin.

The operations of “Parenthesis”, “Square Bracket” and “Curly Bracket”are:

Select some strokes.

Click the icon in the function panel.

Select left parenthesis and press button “OK”.

Select right parenthesis and press button “OK”.

The output data maybe in MathML format, bitmaps or any other format thatmay represent mathematical expressions.

FIGS. 72-85 illustrate another example of a user interface for use withthe handwritten mathematical recognition system. While working in anapplication program, such as Microsoft® Word by Microsoft® Corporationof Redmond, Wash., a user may decide to insert a mathematical expressionby hand. Another program may be built into the application program toallow the user to initiate the insertion. When the user chooses toinsert the handwritten mathematical expression, an associated dialog box7200 may be shown. An example of such a dialog box 7200 is shown in FIG.72.

In this example, the dialog box 7200 is modeless and resizable. Aportion of the dialog box 7200 is handwriting area 7201. To the right ofhandwriting area 7201 are three writing tools: pen 7203, eraser 7205,and clear all 7207. Below handwriting area 7201, an “Initiate” button7209 is separated from the other buttons. When the dialog box isresized, handwriting area 7201 may be resized accordingly; however, theuser interface 7200 may be configured so that the ink and/or the buttonsfor writing tools 7203, 7205, and 7207 do not move or change size. Therelative positions of the buttons and ink may also be configured toremain the same.

When a user activates the pen button 7203, she initiates a writing mode.Similarly, when she activates the eraser button 7205, she initiates anerasing mode. These two modes may be exclusive, i.e., when one is on,the other must be off. When there is no ink in the handwriting area7201, the clear all 7207 button and the Initiate button 7209 may beconfigured to be disabled. When there is ink, these two buttons may beconfigured to be enabled. When the user activates the clear all button7207, all the ink within handwriting area 7201 is removed and the userinitiates a writing mode. When a user activates the Initiate button7209, all the ink is sent to the mathematical expression recognizer andthe user initiates the writing mode.

In accordance with one embodiment, handwriting area 7201 defaults to thewriting mode and is cleared every time the dialog box 7200 is opened. Assuch, no ink is saved. If a user selects an equation that was previouslyentered by handwriting, the dialog box 7200 will not open with theoriginal handwriting filled in. In an alternative embodiment, the userinterface 7200 may be preconfigured and/or allow for a user to configurethe user interface 7200 so that selection of an equation that waspreviously entered in handwriting will open the dialog box 7200 with theoriginal handwriting filled in.

The text “Write equation here and hit Initiate” 7211 may be configuredto appear as a watermark in handwriting area 7201. Such a configurationhelps a user to know where to start. Once the user starts writing, thewatermark is removed and the clear all button 7207 and Initiate button7209 may be enabled, such as shown in FIG. 73.

Once the user finishes writing, she may activate the Initiate button7209 to start the mathematical expression recognition operation asdescribed herein. The system may be configured to show a progress bar7413, such as shown in FIG. 74. The user may stop the recognitionprocess by activating a Stop button 7415. When the user activates theStop button 7415, the recognition system stops and the progress bar 7413goes away. Progress speed in the progress bar 7413 may be estimated bythe number of strokes and other parameters provided by the mathematicalexpression recognizer and/or by any other of a number of differentmethods. When the recognition process is finished, a result display areais shown, with the recognized equation.

As shown in FIG. 75, result display area 7517 is below handwriting area7201. When a recognized equation is first shown 7519, an IP 7521 isplaced at the end of the equation, e.g., a vertical flashing bar, suchas shown. The cursor 7523 in result display area 7517 may be the “I”beam, which is similar to the cursor display position shown in otherapplication programs, such as Microsoft® Word. A user may insert andselect with the IP 7521 in result display area 7517. Common keys on akeyboard, such as arrow keys, backspace, and delete, may operate inresult display area 7517. Three buttons are provided in the resultdisplay area: all symbols 7525, delete 7527, and undo 7529.Corresponding operations and functions of these buttons are describedherein below.

A light gray mask may be applied to one or more portions of the upperzone of dialog box 7500. Such a mask may be used to guide the attentionof the user away from the upper zone and focus on the result displayarea 7517 as correction functionalities may be provided there. A userstill may erase, clear, and rewrite in the handwriting area 7201. Whenthe user moves her cursor position into the upper zone, the mask may beremoved. In such a situation, the cursor may become an arrow. When theuser moves her cursor position into handwriting area 7201, the cursormay become the pen cursor, indicating that the user may write. As soonas the user writes or erases a stroke, result display area 7517 may beemptied or may be collapsed. Otherwise, result display area 7517 stays.In one implementation, when the user activates the Initiate button 7209,a determination is made as to whether there have been any changes to theink since the last time the Initiate button 7209 was activated. If not,the mathematical expression recognizer is not started and the lastrecognized equation is displayed. In another implementation, no matterwhether there have been any changes to the ink, all the ink isrecognized again as if for the first time.

If the user is satisfied with the result, she may activate a Transferbutton 7531 to insert the recognized equation 7519 into the applicationprogram, such as Microsoft® Word. In response, any ink in handwritingarea 7201 is cleared, and the result display area 7517 is collapsed. Thedata sent to the application program may be in a specific type offormat, such as MathML, bitmap, or any other format acceptable by theapplication program. When there is more than one application programopen, the data may be configured to be sent to the application programin focus.

With the correction on ink configuration described above, one problem isthat users may find difficulty understanding ink structure errors, suchas symbol grouping errors. For example, in the equation shown in FIG.75, strokes for the summation sign are not grouped together. One stroke,or group, is recognized as a fraction line, and another stroke, orgroup, is recognized as a “2.”

Although possible to correct this inaccuracy on the ink, it is easierfor a user to identify what is inaccurate in the recognized equation7519. For example, in the equation 7519 shown in FIG. 75, the user maydetermine that the summation sign is missing and that other elementshave been recognized. The user then may delete what is wrong and inserta summation sign.

Besides grouping errors, another common type of error is a layout error,e.g., superscript/subscript relationships and control regions ofdominant symbols, which are recognized inaccurately. Directmanipulation, such as gesture and drag/drop, are simple and convenientways to correct these errors.

Correction on text may include providing candidates, allowing rewriting,enabling drag and drop, and providing editing capabilities. Multiplecandidates are provided for an equation, sub-expressions, and symbols.Users may rewrite part of an equation. Drag and drop allows easy andconvenient correction of layout errors. By providing IP, allowing softkeyboard entry of symbols, and allowing insertion, selection, anddeletion, sufficient editing capabilities ensure all errors may becorrected. In addition to correction user interface provided in resultdisplay area 7517, users may also write and/or erase in handwriting area7201.

The system may be configured to implement “pin” functionality, i.e.,when a user makes a correction to a candidate, the changes are reflectedin other candidates. Or, the system may be configured not to implementthe “pin” functionality, i.e. when a user makes a correction to acandidate, the changes are not respected by other candidates. In such aconfiguration, one implementation may be once users make a correction,such as choosing a candidate from the candidate list, rewrite,insertion, and deletion, candidates for the entire equation will notcontinue to be shown, because the candidates may be far off from whatthe user has corrected so far and may cause user confusion. Similarly,if the correction is inside a sub-expression, candidates for thesub-expression will not continue to be shown. Candidates for a writtensymbol may always be available.

A user may select any part of the recognized equation 7519, as long asthe selection is allowed. If the user activates anywhere else in therecognized equation 7519, selection goes away and IP 7521 is placedwhere clicked.

When a user selects the entire equation, candidates 7635 for theequation 7519 are provided in the dropdown menu 7639. In the exampleshown in FIG. 76, the entire equation 7519 is selected. When the useractivates the dropdown button 7633, candidates 7635 for the equation7519 are shown. The user may choose from the candidates list. Inresponse, the equation in result display area 7517 is replaced by theselected candidate, selection goes away, and IP 7521 is placed at theend of the equation. If the user does not want to choose anything fromthe list, she may choose “Enter Expression Again” 7637 to rewrite theequation. When the user chooses “Enter Expression Again” 7637, it mayperform the same operation as choosing the clear all button 7207, i.e.,all ink is cleared, and result display area 7517 is collapsed.

When a user selects a single character, candidates for the character areprovided in the dropdown menu. In the example shown in FIG. 77, thecharacter “t” 7701 is selected. When the user activates the dropdownbutton 7733 below the character 7701, candidates 7735 for the character7701 are shown. The user may select “+” from the list. In such asituation, character “t” 7701 will be replaced by “+,” selection goesaway, and IP is placed after “+.” If the correct character is not in thelist, the user may rewrite the character in the “Enter Expression Again”area 7737. The “Enter Expression Again” area 7737 is for quick writingto correct errors. For example, the “Enter Expression Again” area 7737may be a fix-sized, without the pen 7203, eraser 7205, and clear all7207 tools. When the user activates an Initiate button 7709, dropdownmenu 7739 stays. The ink that is written is fed to the mathematicalexpression recognizer system. When a result is returned, dropdown menu7739 goes away, and selection is replaced by the recognition result,after which IP is placed.

During the recognition process, a progress bar may be shown. In such asituation, the user may activate a Stop button to stop the recognition.When the user activates the Stop button, dropdown menu 7739 stays, andthe progress bar goes away.

If the user chooses a dominant symbol from the list or the recognitionresult is a single dominant symbol, placeholders for control regionssuch as above fraction line, below fraction line, lower limit, upperlimit, etc., are inserted along with the symbol. As illustrated in FIG.78, placeholders 7851 and 7853 are shown as dotted line boxes.Alternatively, other designs may be used to show placeholders, such asusing a blank area instead of dotted line boxes. The user may place IPin a placeholder 7851, 7853 and insert symbols and/or drag and drop intothe placeholders 7851, 7853. In the example shown in FIG. 78, the userselects the integral sign to replace the character “1.” Placeholders forthe lower limit 7853 and upper limit 7851 are inserted. Locations of theplaceholders are the same as default drop zones, which is describedherein below.

When a user selects a sub-expression, candidates for the sub-expressionare provided in the dropdown menu. In the example shown in FIG. 79, thesub-expression √{square root over (x²+)}y²+C is selected. The useractivates the dropdown button 7933 below the selection. Candidates 7935for the sub-expression are then shown. In this example, no candidate iscorrect. As such, the user may rewrite the sub-expression in area 7937.When the user activates an Initiate button 7909, dropdown menu 7939stays. The ink that is written is fed to the mathematical expressionrecognizer system. When the result is returned, dropdown menu 7939 goesaway, and the selection is replaced by the recognition result, afterwhich IP is placed. During the recognition process, a progress bar maybe shown. In such a situation, the user may activate a Stop button tostop the recognition. When the user activates the Stop button, dropdownmenu 7939 stays, and the progress bar goes away. If the sub-expressionhappens to be a single character, candidates for the character will beshown.

If the selection is neither a character nor a sub-expression, nocandidates are provided in the dropdown menu. The users may rewrite theexpression. In the example shown in FIG. 80, the part$n_{- 1}^{\frac{1{\sigma.}}{2}}$is selected. The selection is neither a character nor a sub-expression.The user may activate dropdown button 8033 below the selection, where nocandidates are shown. The user may rewrite the expression in area 8003.When the user activates an Initiate button 8005, dropdown menu 8039stays. The ink that is written is fed to the mathematical expressionrecognizer system. When a result is returned, dropdown menu 8039 goesaway, and the selection is replaced by the recognition result, afterwhich IP is placed. During the recognition process, a progress bar maybe shown. In such a situation, the user may activate a Stop button tostop the recognition. When the user activates the Stop button, dropdownmenu 8039 stays, and the progress bar goes away.

When there is a selection, a user may drag and drop the selection. Forexample, the user may drag and drop to change subscript/superscriptrelationships, range of a radical sign, range of the numerator anddenominator, etc. Drop locations may be shown in the user interface suchas an “I” beam or shaded boxes shown in FIGS. 81, 82, and 83.Alternatively, other designs may be used to show the drop locations. Inthe example shown in FIG. 81, the superscript +s₁ ⁻nx is selected to bedropped to after m₁, i.e., changed from a superscript to anon-superscript. In the example shown in FIG. 82, the superscript n^(f)is selected to be dropped to the subscript of x, i.e., changed from asuperscript to a subscript. In the example shown in FIG. 83, theexpression y² is selected to be dropped inside the radical sign afterthe character “t”. In this situation, when there is a selection, thedropdown button described above may be shown. However, when the userstarts dragging the selection, the dropdown button goes away.

The drop zones for each character maybe defined. For example, for afraction line, the drop zones are Above, Below, Before and After. For anintegral sign, the drop zones are Upper Limit (to the side of centered),Lower Limit (to the side of centered), Before and After.

The size of the drop zones may be configured based upon any of a numberof different manners, including, but not limited to, the size of theresult display area plus a buffer zone. The buffer zone may include theBefore zone of the first character, the After, Superscript, andSubscript zones of the last character, and the Hat zones of all thecharacters. When there is nothing in a zone, the size of the zone is thebounding box of a single character. A dotted line “I” bar may beconfigured to indicate Before and After drop zones, and a shadedrectangle may be configured to indicate Above, Below, Radicand, Index,Hat, Base, Upper Limit, Lower Limit, Superscript, and Subscript dropzones.

The cursor position may be moved to a location that is the intersectionof several drop zones. Rules may be devised to decide which drop zone toshow. For example,

Show only one zone of a character.

Show at most one dotted line “I” bar.

Show at most one shaded rectangle.

A dotted line “I” bar is always shown inside a shaded rectangle toindicate IP in the shaded rectangle. When there is nothing in a shadedrectangle, a dotted line “I” bar is shown at the beginning of therectangle.

If zone A's parent character is inside zone B, show only zone A.

After the drop, the layout of the recognized equation may need to bechanged. For example, a fraction line may need to be lengthened orshortened or a summation sign may need to be pushed down because it nowhas an upper limit.

Sufficient editing capabilities may be provided to ensure all errors inthe recognized equation may be corrected. Three tools are provided inthe result display area: all symbols, delete, and undo. Their behavioris explained below in Table 2. TABLE 2 Example Behavior of Editing ToolButtons Button Enable/ Label Action Disable all Opens symbol (character)picker. Users choose a Always symbols character and insert. Thecharacter is inserted enabled. at the IP. Note that if there is aselection (which may be multiple characters), the character will replacethe selection. delete At IP, deletes the character to the right. WhenAlways there is selection, deletes selection. enabled. undo Undoprevious action, including candidate Disabled selection, convert,insertion of a single when there is character, deletion, and drag/drop.no previous action.

The symbol picker 8400 may be as simple as a list of all characterssupported by the mathematical expression recognition system.Alternatively, symbol picker 8400 may be implemented in the form of akeyboard with all the characters as buttons on the keyboard. Forexample, symbol picker 8400 may replicate a categorization of allsymbols, excluding those not supported by the mathematical expressionrecognizer system. For example, the categories may include: Algebra,Arrows, Binary Operators, Calculus, Geometry, Greek and LatinCharacters, Operators with Limits, Relational and Logical Operators,Trigonometry, etc. A smaller set of categories may alternatively beemployed. A dropdown menu 8461 may be used to switch between categories,with regular symbols 8463 being listed on the left, while dominantsymbols 8465 are separately listed on the right. Symbol picker 8400 maybe a modeless dialog box that users keep open to insert multiplesymbols. For example, users may dock the dialog box below the textwindow.

When a user clicks on a symbol, it is inserted at the IP. In the exampleshown in FIG. 85, the user inserts the summation sign to replace theselection $n_{- 1}^{\frac{1\sigma}{2}}.$When the user inserts a dominant symbol 8541, placeholders 8551 and 8553for control regions such as above fraction line, below fraction line,lower limit, upper limit, etc., are inserted along with the symbol 8541.Placeholders 8551 and 8553 are shown as dotted line boxes. The user mayplace IP in a placeholder and insert symbols and/or drag and drop intothe placeholders. When there is a placeholder, one way to implement thisis the placeholder becomes the drop zone, i.e., when the drop cursor isinside the placeholder, it becomes a shaded rectangle. When there is aplaceholder and the drop cursor is in the drop zone defined for thecharacter but not inside the placeholder, the drop zone user interfacemay not show.

If the user does not place anything in a placeholder 8551 or 8553, inone implementation, the placeholder is left empty in the result displayarea. The user may select and delete a placeholder 8551 and/or 8553.Locations of lower limit 8553 and upper limit 8551 placeholders arecentered for most operators except for a single integral where locationsof the lower limit and upper limit placeholders are to the side. Asmentioned previously, the user may erase and rewrite in the handwritingarea and have the handwritten equation recognized again.

With respect to an application programming interface (API), variousaspects of the present invention may be provided through an API. Forexample, public APIs may interface with an operating system to allow theoperating system to provide the various features of the presentinvention. In one embodiment, a software architecture stored on one ormore computer-readable media for processing data representative of ahandwritten mathematical expression recognition computation may includea component configured to recognize handwritten mathematical expressionsand an application programming interface to access the component. An APImay receive a request to recognize a handwritten mathematicalexpression, access the necessary function(s) of the recognitioncomponent to perform the operation, and then send the results back to anoperating system. The operating system may use the data provided fromthe API to perform the various features of the present invention.Software applications may also perform various aspects of the presentinvention through APIs in the same way as described in the aboveexample.

While illustrative systems and methods as described herein embodyingvarious aspects of the present invention are shown, it will beunderstood by those skilled in the art, that the invention is notlimited to these embodiments. Modifications may be made by those skilledin the art, particularly in light of the foregoing teachings. Forexample, each of the elements of the aforementioned embodiments may beutilized alone or in combination or sub-combination with elements of theother embodiments. It will also be appreciated and understood thatmodifications may be made without departing from the true spirit andscope of the present invention. The description is thus to be regardedas illustrative instead of restrictive on the present invention.

1. A system for symbol grouping and recognition of a handwrittenmathematical expression, the system comprising: a symbol groupingcomponent configured to group input strokes corresponding to ahandwritten mathematical expression into symbols; and a symbolrecognition component configured to recognize the symbols based uponinformation associated with the grouped input strokes.
 2. The system ofclaim 1, wherein the symbol grouping component is further configured togroup the input strokes into the symbols based upon intra-group andinter-group information associated with the input strokes.
 3. The systemof claim 2, wherein the intra-group information associated with theinput strokes includes a confidence representative of whether the inputstrokes are a certain symbol provided by the symbol recognitioncomponent, and a score representative of whether the input strokes are avalid symbol based upon intra-group spatial information.
 4. The systemof claim 3, wherein the symbol recognition component is furtherconfigured to recognize the input strokes as the certain symbol or anon-symbol and to classify the input strokes with the confidence.
 5. Thesystem of claim 3, wherein the symbol grouping component is furtherconfigured to receive the input strokes, to determine intra-groupspatial information among the input strokes, and to determine whether togroup the input strokes into a symbol based upon the intra-group spatialinformation, wherein the intra-group spatial information includes a sizedifference among the input strokes and an offset measurement among theinput strokes.
 6. The system of claim 3, wherein the symbol groupingcomponent determines whether to group the input strokes into a symbolbased upon dynamic programming, wherein multiple global optimizationcandidates are determined by combining local optimizations.
 7. Thesystem of claim 2, wherein the inter-group information associated withthe input strokes includes spatial relationship information ofneighboring symbols and syntax context.
 8. The system of claim 1,wherein the information associated with the grouped input strokesincludes shape and time series information.
 9. The system of claim 8,wherein the symbol recognition component is further configured toextract off-line features of the input strokes, to classify the off-linefeatures, to provide character candidates with a respective confidencefor the input strokes, and to output the candidates with the respectiveconfidence.
 10. The system of claim 8, wherein the symbol recognitioncomponent is further configured to extract on-line features of the inputstrokes, to classify the on-line features, to provide charactercandidates with a respective confidence for the input strokes, and tooutput the candidates with the respective confidence.
 11. A method forgrouping and recognizing symbols of a handwritten mathematicalexpression, the method comprising: receiving a plurality of inputstrokes corresponding to a handwritten mathematical expression; groupingthe plurality of input strokes into symbols; and recognizing the symbolsbased upon information associated with the grouped input strokes. 12.The method of claim 11, wherein the step of grouping includes groupingbased upon intra-group and inter-group information associated with theplurality of input strokes.
 13. The method of claim 12, furthercomprising steps of: determining a confidence representative of whetherinput strokes of the plurality are a certain symbol provided by a symbolrecognition component; determining a score representative of whether theinput strokes of the plurality are a valid symbol based upon intra-groupspatial information; identifying a spatial relationship between twoneighboring symbols; and identifying a bi-gram probability between thetwo neighboring symbols.
 14. The method of claim 13, further comprisingsteps of: determining whether the input strokes of the plurality is thecertain symbol or a non-symbol; classifying the input strokes of theplurality with the confidence; determining a distance among the inputstrokes of the plurality; determining a size difference among the inputstrokes of the plurality; determining an offset measurement among theinput strokes of the plurality; and determining whether to group theinput strokes of the plurality into a symbol based upon the intra-groupspatial information.
 15. The method of claim 14, further comprisingsteps of: extracting on-line features of the plurality of input strokes;classifying the on-line features; providing character candidates with arespective confidence for the plurality of input strokes; and outputtingthe candidates with the respective confidence.
 16. The method of claim12, further comprising a step of determining whether to group inputstrokes of the plurality into a symbol based upon dynamic programming,wherein multiple global optimization candidates are determined.
 17. Themethod of claim 11, wherein the information associated with the groupedinput strokes includes shape and time series information.
 18. The methodof claim 11, further comprising steps of: extracting off-line featuresof the plurality of input strokes; classifying the off-line features;providing character candidates with a respective confidence for theplurality of input strokes; and outputting the candidates with therespective confidence.
 19. A software architecture stored on one or morecomputer readable media for processing data representative of ahandwritten mathematical expression, comprising: at least one componentconfigured to group and recognize symbols based upon a plurality ofstrokes corresponding to a handwritten mathematical expression; and atleast one application program interface to access the component.
 20. Thesoftware architecture of claim 19, wherein the at least one component isfurther configured to recognize the symbols based upon shape and timeseries information associated with the plurality of strokes.